Papers › A Simple Yet Strong Pipeline for HotpotQA
A Simple Yet Strong Pipeline for HotpotQA
Dirk Groeneveld, Tushar Khot, Mausam, Ashish Sabharwal
State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition. However, does their strong performance on popular multi-hop datasets really justify this added design complexity? Our results suggest that the answer may be no, because even our simple pipeline based on BERT, named Quark, performs surprisingly well. Specifically, on HotpotQA, Quark outperforms these models on both question answering and support identification (and achieves performance very close to a RoBERTa model). Our pipeline has three steps: 1) use BERT to identify potentially relevant sentences independently of each other; 2) feed the set of selected sentences as context into a standard BERT span prediction model to choose an answer; and 3) use the sentence selection model, now with the chosen answer, to produce supporting sentences. The strong performance of Quark resurfaces the importance of carefully exploring simple model designs before using popular benchmarks to justify the value of complex techniques.
In Syntology View this paper on Syntology: its repositories, every harvested function with whether it ran, its licence and the call to fetch it.
Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Question Answering | HotpotQA | Quark + SemanticRetrievalMRS IR | ANS-EM | 0.555 | #37 of 72 | Archive leaderboard | report |
| Question Answering | HotpotQA | Quark + SemanticRetrievalMRS IR | ANS-F1 | 0.675 | #37 of 72 | Archive leaderboard | report |
| Question Answering | HotpotQA | Quark + SemanticRetrievalMRS IR | JOINT-EM | 0.329 | #37 of 72 | Archive leaderboard | report |
| Question Answering | HotpotQA | Quark + SemanticRetrievalMRS IR | JOINT-F1 | 0.562 | #37 of 72 | Archive leaderboard | report |
| Question Answering | HotpotQA | Quark + SemanticRetrievalMRS IR | SUP-EM | 0.456 | #37 of 72 | Archive leaderboard | report |
| Question Answering | HotpotQA | Quark + SemanticRetrievalMRS IR | SUP-F1 | 0.730 | #37 of 72 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections