Papers › Hurdles to Progress in Long-form Question Answering
Hurdles to Progress in Long-form Question Answering
Kalpesh Krishna, Aurko Roy, Mohit Iyyer
The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a detailed analysis reveals several troubling trends: (1) our system's generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train / validation overlap, as at least 81% of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Open-Domain Question Answering | KILT: ELI5 | arxiv.org/abs/2103.06332 | F1 | 22.88 | #3 of 16 | Archive leaderboard | report |
| Open-Domain Question Answering | KILT: ELI5 | arxiv.org/abs/2103.06332 | KILT-F1 | 2.34 | #3 of 16 | Archive leaderboard | report |
| Open-Domain Question Answering | KILT: ELI5 | arxiv.org/abs/2103.06332 | KILT-RL | 2.36 | #3 of 16 | Archive leaderboard | report |
| Open-Domain Question Answering | KILT: ELI5 | arxiv.org/abs/2103.06332 | R-Prec | 10.67 | #3 of 16 | Archive leaderboard | report |
| Open-Domain Question Answering | KILT: ELI5 | arxiv.org/abs/2103.06332 | ROUGE-L | 23.19 | #3 of 16 | Archive leaderboard | report |
| Open-Domain Question Answering | KILT: ELI5 | arxiv.org/abs/2103.06332 | Recall@5 | 24.56 | #3 of 16 | Archive leaderboard | report |
| Question Answering | KILT: ELI5 | c-REALM | F1 | 23.1 | #3 of 7 | Archive leaderboard | report |
| Question Answering | KILT: ELI5 | c-REALM | Rouge-L | 23.4 | #3 of 7 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections