{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/leveraging-large-language-models-for-multiple","title":"Leveraging Large Language Models for Multiple Choice Question Answering","arxiv_id":"2210.12353","date":"2022-10-22","proceeding":null,"authors":["Joshua Robinson","Christopher Michael Rytting","David Wingate"],"abstract":"While large language models (LLMs) like GPT-3 have achieved impressive results on multiple choice question answering (MCQA) tasks in the zero, one, and few-shot settings, they generally lag behind the MCQA state of the art (SOTA). MCQA tasks have traditionally been presented to LLMs like cloze tasks. An LLM is conditioned on a question (without the associated answer options) and its chosen option is the one assigned the highest probability after normalization (for length, etc.). A more natural prompting approach is to present the question and answer options to the LLM jointly and have it output the symbol (e.g., \"A\") associated with its chosen answer option. This approach allows the model to explicitly compare answer options, reduces computational costs, and mitigates the effects of tokenization scheme and answer option representations on answer selection. For the natural approach to be effective, the LLM it is used with must be able to associate answer options with the symbols that represent them. The LLM needs what we term multiple choice symbol binding (MCSB) ability. This ability varies greatly by model. We show that a model with high MCSB ability performs much better with the natural approach than with the traditional approach across 20 diverse datasets and largely closes the gap with the SOTA, suggesting that the MCQA ability of LLMs has been previously underestimated.","url_abs":"https://arxiv.org/abs/2210.12353v3","url_pdf":"https://arxiv.org/pdf/2210.12353v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"leveraging-large-language-models-for-multiple","repo_url":"https://github.com/byu-pccl/leveraging-llms-for-mcqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"answer-selection","task_name":"Answer Selection"},{"task_slug":"multiple-choice-qa","task_name":"Multiple Choice Question Answering (MCQA)"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.12353","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.12353"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/byu-pccl/leveraging-llms-for-mcqa","reach":null}],"summary":{"ran_draft_wrong":3,"ran":3,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"67b75ad99c5cf1ea","entry":"div_dicts","repo":"byu-pccl/leveraging-llms-for-mcqa","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/byu-pccl/leveraging-llms-for-mcqa/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"67b75ad99c5cf1ea"}},{"code_sha256_prefix":"71d3ab2ff4c490ef","entry":"ModelResponseBrown","repo":"byu-pccl/leveraging-llms-for-mcqa","repo_kind":"official","path":"models.py","file_url":"https://github.com/byu-pccl/leveraging-llms-for-mcqa/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"71d3ab2ff4c490ef"}},{"code_sha256_prefix":"2958e1515ddcc220","entry":"ModelResponseNatural","repo":"byu-pccl/leveraging-llms-for-mcqa","repo_kind":"official","path":"models.py","file_url":"https://github.com/byu-pccl/leveraging-llms-for-mcqa/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2958e1515ddcc220"}},{"code_sha256_prefix":"c1b930477c5d6126","entry":"OpenAIModel","repo":"byu-pccl/leveraging-llms-for-mcqa","repo_kind":"official","path":"models.py","file_url":"https://github.com/byu-pccl/leveraging-llms-for-mcqa/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c1b930477c5d6126"}},{"code_sha256_prefix":"f28d8d5956acb8c3","entry":"idx_to_ltr","repo":"byu-pccl/leveraging-llms-for-mcqa","repo_kind":"official","path":"models.py","file_url":"https://github.com/byu-pccl/leveraging-llms-for-mcqa/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f28d8d5956acb8c3"}},{"code_sha256_prefix":"d59587e584260e45","entry":"sub_dicts","repo":"byu-pccl/leveraging-llms-for-mcqa","repo_kind":"official","path":"analyze.py","file_url":"https://github.com/byu-pccl/leveraging-llms-for-mcqa/blob/HEAD/analyze.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d59587e584260e45"}},{"code_sha256_prefix":"c9e7770d2454f7d5","entry":"prep_openai_obj_for_save","repo":"byu-pccl/leveraging-llms-for-mcqa","repo_kind":"official","path":"models.py","file_url":"https://github.com/byu-pccl/leveraging-llms-for-mcqa/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9e7770d2454f7d5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}