{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/artifacts-or-abduction-how-do-llms-answer","title":"Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?","arxiv_id":"2402.12483","date":"2024-02-19","proceeding":null,"authors":["Nishant Balepur","Abhilasha Ravichander","Rachel Rudinger"],"abstract":"Multiple-choice question answering (MCQA) is often used to evaluate large language models (LLMs). To see if MCQA assesses LLMs as intended, we probe if LLMs can perform MCQA with choices-only prompts, where models must select the correct answer only from the choices. In three MCQA datasets and four LLMs, this prompt bests a majority baseline in 11/12 cases, with up to 0.33 accuracy gain. To help explain this behavior, we conduct an in-depth, black-box analysis on memorization, choice dynamics, and question inference. Our key findings are threefold. First, we find no evidence that the choices-only accuracy stems from memorization alone. Second, priors over individual choices do not fully explain choices-only accuracy, hinting that LLMs use the group dynamics of choices. Third, LLMs have some ability to infer a relevant question from choices, and surprisingly can sometimes even match the original question. Inferring the original question is an impressive reasoning strategy, but it cannot fully explain the high choices-only accuracy of LLMs in MCQA. Thus, while LLMs are not fully incapable of reasoning in MCQA, we still advocate for the use of stronger baselines in MCQA benchmarks, the design of robust MCQA datasets for fair evaluations, and further efforts to explain LLM decision-making.","url_abs":"https://arxiv.org/abs/2402.12483v2","url_pdf":"https://arxiv.org/pdf/2402.12483v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"artifacts-or-abduction-how-do-llms-answer","repo_url":"https://github.com/nbalepur/mcqa-artifacts","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"multiple-choice-qa","task_name":"Multiple Choice Question Answering (MCQA)"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.12483","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.12483"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nbalepur/mcqa-artifacts","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":6},"by_repo_kind":{"official":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"f62ccd04ad8f0de6","entry":"check_any_match","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"model/extract_random_questions.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/model/extract_random_questions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f62ccd04ad8f0de6"}},{"code_sha256_prefix":"71e7bc4aa73399ee","entry":"convert_raw_text","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"evaluation/plot_accuracy.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/evaluation/plot_accuracy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71e7bc4aa73399ee"}},{"code_sha256_prefix":"9fb6d123fe0e028c","entry":"eliminate_choice","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"model/prompt.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/model/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fb6d123fe0e028c"}},{"code_sha256_prefix":"bc8d8304271d9ac6","entry":"format_models","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"evaluation/plot_accuracy.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/evaluation/plot_accuracy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bc8d8304271d9ac6"}},{"code_sha256_prefix":"7245cc2c94afad7b","entry":"create_data_choices_even","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"model/data_loader.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/model/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7245cc2c94afad7b"}},{"code_sha256_prefix":"946582a9317efac6","entry":"create_data_choices_even_mmlu","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"model/data_loader.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/model/data_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"946582a9317efac6"}},{"code_sha256_prefix":"20ecd73683b53d64","entry":"generate_text","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"model/run_hf.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/model/run_hf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20ecd73683b53d64"}},{"code_sha256_prefix":"a9ee8ba4a546e084","entry":"load_model","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"model/run_hf.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/model/run_hf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9ee8ba4a546e084"}},{"code_sha256_prefix":"47a4372cd0ec5977","entry":"load_model","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"model/run_hf_question_gen.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/model/run_hf_question_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"47a4372cd0ec5977"}},{"code_sha256_prefix":"9fdf4f61a71db951","entry":"load_model","repo":"nbalepur/mcqa-artifacts","repo_kind":"official","path":"model/run_hf_question_gen_remote.py","file_url":"https://github.com/nbalepur/mcqa-artifacts/blob/HEAD/model/run_hf_question_gen_remote.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9fdf4f61a71db951"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}