Natural Questions

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

5 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 89 papers with code (178 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
  2 Jul 2020 8 repositories listed
  Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge.
• Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering
  10 Nov 2019 7 repositories listed Syntology ran 4 of 6 samples · 2 unverified · 4 pointer-only (licence)
  We introduce an approach for open-domain question answering (QA) that retrieves and reads a passage graph, where vertices are passages of text and edges represent relationships that are derived from an external…
• RealFormer: Transformer Likes Residual Attention
  21 Dec 2020 5 repositories listed
  Transformer is the backbone of modern NLP models.
• Relevance-guided Supervision for OpenQA with ColBERT
  1 Jul 2020 5 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages.
• ST-MoE: Designing Stable and Transferable Sparse Expert Models
  17 Feb 2022 3 repositories listed Syntology ran 5 of 5 samples · 0 unverified · 5 pointer-only (licence)
  But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning.
• TempoQR: Temporal Question Reasoning over Knowledge Graphs
  10 Dec 2021 3 repositories listed
  The first computes a textual representation of a given question, the second combines it with the entity embeddings for entities involved in the question, and the third generates question-specific time embeddings.
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
  8 Dec 2021 3 repositories listed
  Language modelling provides a step towards intelligent communication systems by harnessing large repositories of written human knowledge to better predict and understand the world.
• AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Data
  9 Oct 2020 3 repositories listed Syntology ran 12 of 26 samples · 14 unverified · 26 pointer-only (licence)
  To demonstrate the generality of AutoQA, we also apply it to the Overnight dataset.
• Event Extraction by Answering (Almost) Natural Questions
  28 Apr 2020 3 repositories listed Syntology ran 2 of 3 samples · 1 unverified
  The problem of event extraction requires detecting the event trigger and extracting its corresponding arguments.
• A BERT Baseline for the Natural Questions
  24 Jan 2019 3 repositories listed Syntology ran 0 of 10 samples · 10 unverified
  This technical note describes a new baseline for the Natural Questions.
• Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval
  7 Apr 2025 2 repositories listed
  Generative information retrieval (GenIR) is a promising neural retrieval paradigm that formulates document retrieval as a document identifier (docid) generation task, allowing for end-to-end optimization toward a…
• Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness
  21 Jan 2023 2 repositories listed
  Neural text-to-SQL models have achieved remarkable performance in translating natural language questions into SQL queries.
• DPTDR: Deep Prompt Tuning for Dense Passage Retrieval
  24 Aug 2022 2 repositories listed
  We believe this work facilitates the industry, as it saves enormous efforts and costs of deployment and increases the utility of computing resources.
• Atlas: Few-shot Learning with Retrieval Augmented Language Models
  5 Aug 2022 2 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  Retrieval augmented models are known to excel at knowledge intensive tasks without the need for as many parameters, but it is unclear whether they work in few-shot settings.
• Would You Ask it that Way? Measuring and Improving Question Naturalness for Knowledge Graph Question Answering
  25 May 2022 2 repositories listed
  The construction of this test collection also sheds light on the challenges of constructing large-scale KGQA datasets with genuinely NL questions.
• QAMPARI: An Open-domain Question Answering Benchmark for Questions with Many Answers from Multiple Paragraphs
  25 May 2022 2 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  Our results highlight the need for developing ODQA models that handle a broad range of question types, including single and multi-answer questions.
• Continual Learning with Knowledge Transfer for Sentiment Classification
  18 Dec 2021 2 repositories listed
  In this setting, the CL system learns a sequence of SC tasks incrementally in a neural network, where each task builds a classifier to classify the sentiment of reviews of a particular product category or domain.
• End-to-End Training of Neural Retrievers for Open-Domain Question Answering
  2 Jan 2021 2 repositories listed
  We also explore two approaches for end-to-end supervised training of the reader and retriever components in OpenQA models.
• MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering
  30 Jul 2020 2 repositories listed Syntology ran 0 of 5 samples · 5 unverified
  Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets.
• TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection
  11 Nov 2019 2 repositories listed
  Additionally, we show that the transfer step of TANDA makes the adaptation step more robust to noise.
• Generating Natural Questions About an Image
  19 Mar 2016 2 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  There has been an explosion of work in the vision & language community during the past few years from image captioning to video transcription, and answering questions about images.
• Constructing and Evaluating Declarative RAG Pipelines in PyTerrier
  12 Jun 2025 1 repository listed
  Search engines often follow a pipeline architecture, where complex but effective reranking components are used to refine the results of an initial retrieval.
• PoisonArena: Uncovering Competing Poisoning Attacks in Retrieval-Augmented Generation
  18 May 2025 1 repository listed
  Retrieval-Augmented Generation (RAG) systems, widely used to improve the factual grounding of large language models (LLMs), are increasingly vulnerable to poisoning attacks, where adversaries inject manipulated content…
• GRADA: Graph-based Reranker against Adversarial Documents Attack
  12 May 2025 1 repository listed
  Retrieval Augmented Generation (RAG) frameworks improve the accuracy of large language models (LLMs) by integrating external knowledge from retrieved documents, thereby overcoming the limitations of models' static…
• Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition
  12 May 2025 1 repository listed
  Recent research has questioned the role of fine-tuning vs.
• S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability of Large Reasoning Models
  14 Apr 2025 1 repository listed
  We introduce S1-Bench, a novel benchmark designed to evaluate the performance of Large Reasoning Models (LRMs) on simple tasks that favor intuitive system 1 thinking rather than deliberative system 2 reasoning.
• DragonVerseQA: Open-Domain Long-Form Context-Aware Question-Answering
  21 Dec 2024 1 repository listed
  This paper proposes a novel approach to develop an open-domain and long-form Over-The-Top (OTT) Question-Answering (QA) dataset, DragonVerseQA, specifically oriented to the fantasy universe of "House of the Dragon" and…
• ELOQ: Resources for Enhancing LLM Detection of Out-of-Scope Questions
  18 Oct 2024 1 repository listed
  Retrieval-augmented generation (RAG) has become integral to large language models (LLMs), particularly for conversational AI systems where user questions may reference knowledge beyond the LLMs' training cutoff.
• FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection
  12 Aug 2024 1 repository listed Syntology ran 0 of 2 samples · 2 unverified · 2 pointer-only (licence)
  Open Domain Question Answering (ODQA) has been advancing rapidly in recent times, driven by significant developments in dense passage retrieval and pretrained language models.
• LLaSA: A Multimodal LLM for Human Activity Analysis Through Wearable and Smartphone Sensors
  20 Jun 2024 1 repository listed Syntology ran 6 of 7 samples · 1 unverified · 7 pointer-only (licence)
  Wearables generate rich motion data, yet current systems only classify what happened - failing to support natural questions about why it happened or what it means.

Syntology lines on 12 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.