Multiple Choice Question Answering (MCQA)

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

31 leaderboard tables shown for this task, 31 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 31 until expanded.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

9 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 37 papers with code (65 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Llama 2: Open Foundation and Fine-Tuned Chat Models
  18 Jul 2023 19 repositories listed Syntology ran 31 of 52 samples · 21 unverified · 16 pointer-only (licence)
  In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.
• PaLM: Scaling Language Modeling with Pathways
  5 Apr 2022 7 repositories listed Syntology ran 30 of 37 samples · 7 unverified
  To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM.
• From Recognition to Cognition: Visual Commonsense Reasoning
  27 Nov 2018 4 repositories listed Syntology ran 0 of 6 samples · 6 unverified
  While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world.
• QuALITY: Question Answering with Long Input Texts, Yes!
  16 Dec 2021 3 repositories listed
  To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5, 000 tokens, much longer…
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
  8 Dec 2021 3 repositories listed
  Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.
• Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering
  28 Aug 2018 3 repositories listed
  In this paper we propose a retriever-reader model that learns to attend on essential terms during the question answering process.
• BloombergGPT: A Large Language Model for Finance
  30 Mar 2023 2 repositories listed
  The use of NLP in the realm of financial technology is broad and complex, with applications ranging from sentiment analysis and named entity recognition to question answering.
• Variational Open-Domain Question Answering
  23 Sep 2022 2 repositories listed Syntology ran 0 of 1 samples · 1 unverified
  Retrieval-augmented models have proven to be effective in natural language processing tasks, yet there remains a lack of research on their optimization using variational inference.
• Training Compute-Optimal Large Language Models
  29 Mar 2022 2 repositories listed Syntology ran 8 of 11 samples · 3 unverified · 4 pointer-only (licence)
  We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
• MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
  1 Oct 2019 2 repositories listed
  Machine Reading Comprehension (MRC) for question answering (QA), which aims to answer a question given the relevant context passages, is an important way to test the ability of intelligence systems to understand human…
• Question-Aware Knowledge Graph Prompting for Enhancing Large Language Models
  30 Mar 2025 1 repository listed
  Recent approaches leverage Graph Neural Networks (GNNs) to generate KG-based input embedding prefixes as soft prompts for LLMs but fail to account for question relevance, resulting in noisy prompts.
• Wrong Answers Can Also Be Useful: PlausibleQA -- A Large-Scale QA Dataset with Answer Plausibility Scores
  22 Feb 2025 1 repository listed
  As by their design, LLMs prioritize generating correct answers, the value of highly plausible yet incorrect answers (candidate answers) tends to be overlooked.
• Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning
  8 Feb 2025 1 repository listed
  We then develop an LLM-based automated evaluation framework to identify reasoning errors and evaluate the performance of LLMs.
• MedG-KRP: Medical Graph Knowledge Representation Probing
  14 Dec 2024 1 repository listed
  We test GPT-4, Llama3-70b, and PalmyraMed-70b, a specialized medical model.
• KnowledgePrompts: Exploring the Abilities of Large Language Models to Solve Proportional Analogies via Knowledge-Enhanced Prompting
  1 Dec 2024 1 repository listed
  Making analogies is fundamental to cognition.
• Differentiating Choices via Commonality for Multiple-Choice Question Answering
  21 Aug 2024 1 repository listed
  Our model captures token-level attention of each choice to the question, and separates tokens of the question attended to by all the choices (i.
• EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning
  13 May 2024 1 repository listed Syntology ran 7 of 10 samples · 3 unverified
  In this paper, we introduce EconLogicQA, a rigorous benchmark designed to assess the sequential reasoning capabilities of large language models (LLMs) within the intricate realms of economics, business, and supply chain…
• AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts
  1 May 2024 1 repository listed Syntology ran 1 of 2 samples · 1 unverified
  We introduce AdaMoLE, a novel method for fine-tuning large language models (LLMs) through an Adaptive Mixture of Low-Rank Adaptation (LoRA) Experts.
• From Multiple-Choice to Extractive QA: A Case Study for English and Arabic
  26 Apr 2024 1 repository listed
  The rapid evolution of Natural Language Processing (NLP) has favoured major languages such as English, leaving a significant gap for many others due to limited resources.
• Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?
  19 Feb 2024 1 repository listed Syntology ran 4 of 10 samples · 6 unverified · 10 pointer-only (licence)
  Inferring the original question is an impressive reasoning strategy, but it cannot fully explain the high choices-only accuracy of LLMs in MCQA.
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
  27 Nov 2023 1 repository listed Syntology ran 9 of 14 samples · 5 unverified
  Large language models (LLMs) can potentially democratize access to medical knowledge.
• Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations
  2 Oct 2023 1 repository listed Syntology ran 4 of 5 samples · 1 unverified · 5 pointer-only (licence)
  Large language and vision-language models are rapidly being deployed in practice thanks to their impressive capabilities in instruction following, in-context learning, and so on.
• BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine
  18 Aug 2023 1 repository listed
  BioMedGPT aligns different biological modalities with natural language via a large generative language model, namely, BioMedGPT-LM.
• M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Models
  17 May 2023 1 repository listed
  Comprehensively evaluating the capability of large language models in multiple tasks is of great importance.
• Towards Expert-Level Medical Question Answering with Large Language Models
  16 May 2023 1 repository listed
  Large language models (LLMs) have catalyzed significant progress in medical question answering; Med-PaLM was the first model to exceed a "passing" score in US Medical Licensing Examination (USMLE) style questions with a…
• FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain
  9 Apr 2023 1 repository listed
  This paper introduces FrenchMedMCQA, the first publicly available Multiple-Choice Question Answering (MCQA) dataset in French for medical domain.
• Large Language Models Encode Clinical Knowledge
  26 Dec 2022 1 repository listed
  To resolve this we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars.
• Galactica: A Large Language Model for Science
  16 Nov 2022 1 repository listed Syntology ran 0 of 2 samples · 2 unverified
  We believe these results demonstrate the potential for language models as a new interface for science.
• Leveraging Large Language Models for Multiple Choice Question Answering
  22 Oct 2022 1 repository listed Syntology ran 6 of 7 samples · 1 unverified
  A more natural prompting approach is to present the question and answer options to the LLM jointly and have it output the symbol (e.
• Can large language models reason about medical questions?
  17 Jul 2022 1 repository listed Syntology ran 0 of 10 samples · 10 unverified
  Although large language models (LLMs) often produce impressive outputs, it remains unclear how they perform in real-world scenarios requiring strong reasoning skills and expert domain knowledge.

Syntology lines on 13 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.