Browse State-of-the-Art › Multiple-choice
Multiple-choice
483 papers with code · 0 benchmarks · 12 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
No benchmark for this task in the archive.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
12 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
No subtask under this task in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 1,107 · with a code link: 483 · where Syntology ran a sample: 161 (116 with a run with no instrument failure, 45 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (161 of 1,107 tagged: 116 with a run with no instrument failure, 45 where every run was a failure of Syntology's instrument)
Page 1 of 12: papers 1 to 100 of 1,107, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
3 May 2015 21 repositories listed Syntology 6 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 7 harvested samples) · 6 pointer-only (licence)Given an image and a natural language question about the image, the task is to provide an accurate natural language answer.
-
30 Jan 2023 17 repositories listed Syntology community repositories only · 4 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 4 unverified; every one of the 4 samples that ran constructed an object rather than computing a result (of 8 harvested samples) · 1 pointer-only (licence)The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models.
-
16 Nov 2023 6 repositories listed Syntology community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 2 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 7 harvested samples) · 2 pointer-only (licence)In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM.
-
29 Dec 2022 5 repositories listedNearly all jurisdictions in the United States require a professional license exam, commonly referred to as "the Bar Exam," as a precondition for law practice.
-
29 Apr 2022 5 repositories listed Syntology 18 ran (of which 6 constructed an object rather than computing a result; 11 with no instrument failure: 0 honoured, 1 violated, 10 with no contract checked; 7 where Syntology's instrument failed) · 6 unverified (of 24 harvested samples) · 8 pointer-only (licence)Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research.
-
5 Jun 2024 4 repositories listedIn this study, we introduce a novel approach to generate quizzes from Turkish educational texts, marking a pioneering endeavor in educational technology specifically tailored to the Turkish educational context.
-
22 Feb 2024 4 repositories listed Syntology official (archive's flag): 10 ran · 25 ran (of which 0 constructed an object rather than computing a result; 22 with no instrument failure: 4 honoured, 1 violated, 17 with no contract checked; 3 where Syntology's instrument failed) · 14 unverified (of 39 harvested samples) · 17 pointer-only (licence)The versatility of large language models (LLMs) led to the creation of diverse benchmarks that thoroughly test a variety of language models' abilities.
-
9 Dec 2023 4 repositories listed Syntology official (archive's flag): 13 ran · 21 ran (of which 0 constructed an object rather than computing a result; 16 with no instrument failure: 4 honoured, 0 violated, 12 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (of 22 harvested samples) · 10 pointer-only (licence)We introduce Contrastive Activation Addition (CAA), an innovative method for steering language models by modifying their activations during forward passes.
-
3 Nov 2023 4 repositories listedOne of the major limitations of the basic score is that it treats all words as equally important.
-
27 Nov 2018 4 repositories listed Syntology 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world.
-
2 Nov 2018 4 repositories listed Syntology community repositories only · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)To investigate question answering with prior knowledge, we present CommonsenseQA: a challenging new dataset for commonsense question answering.
-
31 Dec 2024 3 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)To bridge this gap, we introduce MapEval, a benchmark designed to assess diverse and complex map-based user queries with geo-spatial reasoning.
-
11 Jun 2024 3 repositories listed Syntology community repositories only · 12 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 1 honoured, 1 violated, 5 with no contract checked; 5 where Syntology's instrument failed) · 5 unverified (of 17 harvested samples) · 8 pointer-only (licence)In this paper, we present the VideoLLaMA 2, a set of Video Large Language Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio understanding in video and audio-oriented tasks.
-
28 Nov 2023 3 repositories listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 1 violated, 2 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (of 10 harvested samples) · 2 pointer-only (licence)With the rapid development of Multi-modal Large Language Models (MLLMs), a number of diagnostic benchmarks have recently emerged to evaluate the comprehension capabilities of these models.
-
20 Nov 2023 3 repositories listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 8 harvested samples)We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.
-
30 Jul 2023 3 repositories listedBased on powerful Large Language Models (LLMs), recent generative Multimodal Large Language Models (MLLMs) have gained prominence as a pivotal research area, exhibiting remarkable capability for both comprehension and…
-
12 Jul 2023 3 repositories listedIn response to these challenges, we propose MMBench, a bilingual benchmark for assessing the multi-modal capabilities of VLMs.
-
27 May 2023 3 repositories listed Syntology official (archive's flag): 8 ran · 10 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 4 where Syntology's instrument failed) · 7 unverified (of 17 harvested samples) · 4 pointer-only (licence)Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory.
-
16 Dec 2021 3 repositories listedTo enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5, 000 tokens, much longer…
-
28 Sep 2020 3 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community.
-
28 Aug 2018 3 repositories listedIn this paper we propose a retriever-reader model that learns to attend on essential terms during the question answering process.
-
27 Jun 2016 3 repositories listedVisual question answering (VQA) is an interesting learning setting for evaluating the abilities and shortcomings of current systems for image understanding.
-
20 May 2025 2 repositories listed Syntology 12 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 6 where Syntology's instrument failed) · 2 unverified (of 14 harvested samples) · 5 pointer-only (licence)Large multimodal models (LMMs) have recently emerged as a powerful tool for long video understanding (LVU), prompting the development of standardized LVU benchmarks to evaluate their performance.
-
28 Feb 2025 2 repositories listed Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 9 harvested samples)Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research.
-
17 Nov 2024 2 repositories listedThe advancement of Multimodal Large Language Models (MLLMs) has enabled significant progress in multimodal understanding, expanding their capacity to analyze video content.
-
6 Aug 2024 2 repositories listedWe present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series.
-
3 Aug 2024 2 repositories listed Syntology official (archive's flag): 9 ran · 11 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 2 violated, 2 with no contract checked; 6 where Syntology's instrument failed) · 3 unverified (of 14 harvested samples) · 6 pointer-only (licence)The recent surge of Multimodal Large Language Models (MLLMs) has fundamentally reshaped the landscape of AI research and industry, shedding light on a promising path toward the next AI milestone.
-
2 Aug 2024 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music understanding in multimodal language models focused on audio.
-
15 Jul 2024 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples) · 4 pointer-only (licence)We demonstrate that an attacker can embed a backdoor in LLMs, which, when activated by a specific trigger in the input, manipulates the model's uncertainty without affecting the final output.
-
14 Jun 2024 2 repositories listedEnhancing Language Models' (LMs) ability to understand purchase intentions in E-commerce scenarios is crucial for their effective assistance in various downstream tasks.
-
17 May 2024 2 repositories listed
-
SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr 2024 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
11 Apr 2024 2 repositories listed
-
4 Apr 2024 2 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples)
-
5 Mar 2024 2 repositories listed Syntology 3 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
20 Feb 2024 2 repositories listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 6 where Syntology's instrument failed) · 4 unverified (of 14 harvested samples)
-
15 Feb 2024 2 repositories listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 4 where Syntology's instrument failed) · 1 unverified (of 6 harvested samples)
-
6 Feb 2024 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
31 Jan 2024 2 repositories listed Syntology official (archive's flag): 1 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples) · 1 pointer-only (licence)
-
28 Nov 2023 2 repositories listed
-
10 Nov 2023 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
8 Oct 2023 2 repositories listed Syntology 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 3 pointer-only (licence)
-
31 Aug 2023 2 repositories listed Syntology official: harvested, nothing ran · 0 ran · 6 unverified (of 6 harvested samples) · 6 pointer-only (licence)
-
30 Jul 2023 2 repositories listed
-
26 Jun 2023 2 repositories listed
-
9 Jun 2023 2 repositories listed
-
7 May 2023 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
5 May 2023 2 repositories listed
-
28 Mar 2023 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample)
-
28 Jan 2023 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
23 Sep 2022 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
28 Jun 2022 2 repositories listed
-
13 Jan 2022 2 repositories listed Syntology official (archive's flag): 2 ran · 13 ran (of which 8 constructed an object rather than computing a result; 9 with no instrument failure: 0 honoured, 0 violated, 9 with no contract checked; 4 where Syntology's instrument failed) · 11 unverified (of 24 harvested samples) · 6 pointer-only (licence)
-
6 Sep 2021 2 repositories listed
-
19 Apr 2021 2 repositories listed
-
18 Apr 2021 2 repositories listed Syntology official (archive's flag): 1 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 9 harvested samples)
-
16 Apr 2021 2 repositories listed Syntology community repositories only · 5 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 5 where Syntology's instrument failed) · 0 unverified (of 5 harvested samples)
-
18 Feb 2021 2 repositories listed
-
1 Oct 2020 2 repositories listed
-
2 May 2020 2 repositories listed Syntology official (archive's flag): 4 ran · 4 ran (of which 3 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 1 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 7 harvested samples) · 1 pointer-only (licence)
-
23 Dec 2019 2 repositories listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
1 Oct 2019 2 repositories listed
-
15 Aug 2019 2 repositories listed Syntology 14 ran (of which 0 constructed an object rather than computing a result; 11 with no instrument failure: 0 honoured, 0 violated, 11 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 15 harvested samples) · 2 pointer-only (licence)
-
8 Sep 2018 2 repositories listed
-
12 Jun 2017 2 repositories listed
-
14 Jun 2025 1 repository listed
-
6 Jun 2025 1 repository listed
-
4 Jun 2025 1 repository listed
-
2 Jun 2025 1 repository listed
-
30 May 2025 1 repository listed
-
30 May 2025 1 repository listed
-
29 May 2025 1 repository listed
-
29 May 2025 1 repository listed
-
29 May 2025 1 repository listed
-
25 May 2025 1 repository listed
-
23 May 2025 1 repository listed Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 2 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
22 May 2025 1 repository listed
-
18 May 2025 1 repository listed
-
18 May 2025 1 repository listed
-
16 May 2025 1 repository listed
-
16 May 2025 1 repository listed
-
16 May 2025 1 repository listed
-
16 May 2025 1 repository listed
-
13 May 2025 1 repository listed
-
13 May 2025 1 repository listed
-
13 May 2025 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 2 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
13 May 2025 1 repository listed
-
12 May 2025 1 repository listed
-
7 May 2025 1 repository listed
-
6 May 2025 1 repository listed
-
6 May 2025 1 repository listed
-
2 May 2025 1 repository listed
-
20 Apr 2025 1 repository listed
-
9 Apr 2025 1 repository listed
-
7 Apr 2025 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
3 Apr 2025 1 repository listed
-
31 Mar 2025 1 repository listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 3 honoured, 0 violated, 4 with no contract checked; 0 where Syntology's instrument failed) · 2 unverified (of 9 harvested samples)
-
30 Mar 2025 1 repository listed
Syntology lines on 45 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections