Browse State-of-the-Art › Multiple Choice Question Answering (MCQA)
Multiple Choice Question Answering (MCQA)
37 papers with code · 31 benchmarks · 9 datasets archive 2025-07-28
A multiple-choice question (MCQ) is composed of two parts: a stem that identifies the question or problem, and a set of alternatives or possible answers that contain a key that is the best answer to the question, and a number of distractors that are plausible but incorrect answers to the question.
In a k-way MCQA task, a model is provided with a question q, a set of candidate options O = {O1, . . . , Ok}, and a supporting context for each option C = {C1, . . . , Ck}. The model needs to predict the correct answer option that is best supported by the given contexts.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
31 leaderboard tables shown for this task, 31 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 31 until expanded.
Syntology 12 newer papers were judged to report on this task's benchmarks so far (Claude Sonnet 4.5, reading their tables: a model's judgement, not a result; 0 placed): MedMCQA (2 judged, 0 placed), BIG-bench (Hyperbaton) (1 judged, 0 placed), BIG-bench (Movie Recommendation) (1 judged, 0 placed), MMLU (College Biology) (5 judged, 0 placed), MMLU (Medical Genetics) (3 judged, 0 placed), MMLU (Professional medicine) (1 judged, 0 placed), MMLU (Formal Logic) (1 judged, 0 placed), MMLU (Abstract Algebra) (2 judged, 0 placed), MMLU (Econometrics) (1 judged, 0 placed), MMLU (High School Computer Science) (1 judged, 0 placed), MMLU (College Mathematics) (4 judged, 0 placed), MMLU (Astronomy) (1 judged, 0 placed). Drawn from 8,886 of 18,637 newer papers checked so far (733 more had nothing to place; 41 could not be read).
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
9 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 65 · with a code link: 37 · where Syntology ran a sample: 16 (13 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (16 of 65 tagged: 13 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 65 of 65, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
18 Jul 2023 19 repositories listed Syntology community repositories only · 33 ran (of which 7 constructed an object rather than computing a result; 20 with no instrument failure: 1 honoured, 1 violated, 18 with no contract checked; 13 where Syntology's instrument failed) · 19 unverified (of 52 harvested samples) · 20 pointer-only (licence)In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.
-
5 Apr 2022 7 repositories listed Syntology 32 ran (of which 16 constructed an object rather than computing a result; 24 with no instrument failure: 2 honoured, 1 violated, 21 with no contract checked; 8 where Syntology's instrument failed) · 5 unverified (of 37 harvested samples)To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
-
27 Nov 2018 4 repositories listed Syntology 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world.
-
16 Dec 2021 3 repositories listedTo enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5, 000 tokens, much longer…
-
8 Dec 2021 3 repositories listedLanguage modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.
-
28 Aug 2018 3 repositories listedIn this paper we propose a retriever-reader model that learns to attend on essential terms during the question answering process.
-
30 Mar 2023 2 repositories listedThe use of NLP in the realm of financial technology is broad and complex, with applications ranging from sentiment analysis and named entity recognition to question answering.
-
23 Sep 2022 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)Retrieval-augmented models have proven to be effective in natural language processing tasks, yet there remains a lack of research on their optimization using variational inference.
-
29 Mar 2022 2 repositories listed Syntology 8 ran (of which 3 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 11 harvested samples) · 4 pointer-only (licence)We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
-
1 Oct 2019 2 repositories listedMachine Reading Comprehension (MRC) for question answering (QA), which aims to answer a question given the relevant context passages, is an important way to test the ability of intelligence systems to understand human…
-
30 Mar 2025 1 repository listedRecent approaches leverage Graph Neural Networks (GNNs) to generate KG-based input embedding prefixes as soft prompts for LLMs but fail to account for question relevance, resulting in noisy prompts.
-
22 Feb 2025 1 repository listedAs by their design, LLMs prioritize generating correct answers, the value of highly plausible yet incorrect answers (candidate answers) tends to be overlooked.
-
8 Feb 2025 1 repository listedWe then develop an LLM-based automated evaluation framework to identify reasoning errors and evaluate the performance of LLMs.
-
14 Dec 2024 1 repository listedWe test GPT-4, Llama3-70b, and PalmyraMed-70b, a specialized medical model.
-
1 Dec 2024 1 repository listedMaking analogies is fundamental to cognition.
-
21 Aug 2024 1 repository listedOur model captures token-level attention of each choice to the question, and separates tokens of the question attended to by all the choices (i. e., commonalities) from those by individual choices (i. e., nuances).
-
13 May 2024 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 1 honoured, 0 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (of 10 harvested samples) · 2 pointer-only (licence)In this paper, we introduce EconLogicQA, a rigorous benchmark designed to assess the sequential reasoning capabilities of large language models (LLMs) within the intricate realms of economics, business, and supply chain…
-
1 May 2024 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)We introduce AdaMoLE, a novel method for fine-tuning large language models (LLMs) through an Adaptive Mixture of Low-Rank Adaptation (LoRA) Experts.
-
26 Apr 2024 1 repository listedThe rapid evolution of Natural Language Processing (NLP) has favoured major languages such as English, leaving a significant gap for many others due to limited resources.
-
19 Feb 2024 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 6 unverified (of 10 harvested samples) · 10 pointer-only (licence)Inferring the original question is an impressive reasoning strategy, but it cannot fully explain the high choices-only accuracy of LLMs in MCQA.
-
27 Nov 2023 1 repository listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified (of 14 harvested samples)Large language models (LLMs) can potentially democratize access to medical knowledge.
-
2 Oct 2023 1 repository listed Syntology official (archive's flag): 3 ran · 4 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 5 harvested samples) · 5 pointer-only (licence)Large language and vision-language models are rapidly being deployed in practice thanks to their impressive capabilities in instruction following, in-context learning, and so on.
-
18 Aug 2023 1 repository listedBioMedGPT aligns different biological modalities with natural language via a large generative language model, namely, BioMedGPT-LM.
-
17 May 2023 1 repository listedComprehensively evaluating the capability of large language models in multiple tasks is of great importance.
-
16 May 2023 1 repository listedLarge language models (LLMs) have catalyzed significant progress in medical question answering; Med-PaLM was the first model to exceed a "passing" score in US Medical Licensing Examination (USMLE) style questions with a…
-
9 Apr 2023 1 repository listedThis paper introduces FrenchMedMCQA, the first publicly available Multiple-Choice Question Answering (MCQA) dataset in French for medical domain.
-
26 Dec 2022 1 repository listedTo resolve this we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars.
-
16 Nov 2022 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)We believe these results demonstrate the potential for language models as a new interface for science.
-
22 Oct 2022 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 3 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples)A more natural prompting approach is to present the question and answer options to the LLM jointly and have it output the symbol (e. g., "A") associated with its chosen answer option.
-
17 Jul 2022 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 10 harvested samples)Although large language models (LLMs) often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.
-
30 Apr 2022 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 6 unverified (of 11 harvested samples)
-
27 Mar 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)
-
29 Jan 2022 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
3 Oct 2021 1 repository listed Syntology official: not harvested · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
22 Sep 2021 1 repository listed
-
8 Nov 2020 1 repository listed
-
12 Oct 2020 1 repository listed
-
CP-Router: An Uncertainty-Aware Router Between LLM and LRM26 May 2025 0 repositories listed
-
Improving LLM First-Token Predictions in Multiple-Choice Question Answering via Prefilling Attack21 May 2025 0 repositories listed
-
Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Information9 May 2025 0 repositories listed
-
Correctness Coverage Evaluation for Medical Multiple-Choice Question Answering Based on the Enhanced Conformal Prediction Framework7 Mar 2025 0 repositories listed
-
Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learning27 Feb 2025 0 repositories listed
-
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above19 Feb 2025 0 repositories listed
-
First Token Probability Guided RAG for Telecom Question Answering11 Jan 2025 0 repositories listed
-
LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering13 Dec 2024 0 repositories listed
-
SandboxAQ's submission to MRL 2024 Shared Task on Multi-lingual Multi-task Information Retrieval28 Oct 2024 0 repositories listed
-
Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models18 Oct 2024 0 repositories listed
-
14 Jun 2024 0 repositories listed
-
Rethinking Generative Large Language Model Evaluation for Semantic Comprehension12 Mar 2024 0 repositories listed
-
KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations3 Mar 2024 0 repositories listed
-
Unsupervised multiple choices question answering via universal corpus27 Feb 2024 0 repositories listed
-
LLMs May Perform MCQA by Selecting the Least Incorrect Option2 Feb 2024 0 repositories listed
-
Evaluating the Symbol Binding Ability of Large Language Models for Multiple-Choice Questions in Vietnamese General Education18 Oct 2023 0 repositories listed
-
Generating multiple-choice questions for medical question answering with distractors and cue-masking13 Mar 2023 0 repositories listed
-
HRCA+: Advanced Multiple-choice Machine Reading Comprehension Method1 Jun 2022 0 repositories listed
-
Context-guided Triple Matching for Multiple Choice Question Answering16 Jan 2022 0 repositories listed
-
Disaggregating Hops: Can We Guide a Multi-Hop Reasoning Language Model to Incrementally Learn at each Hop?16 Jan 2022 0 repositories listed
-
Transliteration: A Simple Technique For Improving Multilingual Language Modeling29 Sep 2021 0 repositories listed
-
Context-guided Triple Matching for Multiple Choice Question Answering27 Sep 2021 0 repositories listed
-
Fine-tuning BERT with Focus Words for Explanation Regeneration1 Dec 2020 0 repositories listed
-
What do we expect from Multiple-choice QA Systems?20 Nov 2020 0 repositories listed
-
Context Modeling with Evidence Filter for Multiple Choice Question Answering6 Oct 2020 0 repositories listed
-
Multi-source Meta Transfer for Low Resource Multiple-Choice Question Answering1 Jul 2020 0 repositories listed
-
11 Nov 2015 0 repositories listed
Syntology lines on 18 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections