Papers › Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection
Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection
Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti
An important task for designing QA systems is answer sentence selection (AS2): selecting the sentence containing (or constituting) the answer to a question from a set of retrieved relevant documents. In this paper, we propose three novel sentence-level transformer pre-training objectives that incorporate paragraph-level semantics within and across documents, to improve the performance of transformers for AS2, and mitigate the requirement of large labeled datasets. Specifically, the model is tasked to predict whether: (i) two sentences are extracted from the same paragraph, (ii) a given sentence is extracted from a given paragraph, and (iii) two paragraphs are extracted from the same document. Our experiments on three public and one industrial AS2 datasets demonstrate the empirical superiority of our pre-trained transformers over baseline models such as RoBERTa and ELECTRA for AS2.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Answer Selection | ASNQ | DeBERTa-V3-Large + SSP | MAP | 0.743 | #1 of 3 | Archive leaderboard | report |
| Answer Selection | ASNQ | DeBERTa-V3-Large + SSP | MRR | 0.800 | #1 of 3 | Archive leaderboard | report |
| Answer Selection | ASNQ | ELECTRA-Base + SSP | MAP | 0.697 | #2 of 3 | Archive leaderboard | report |
| Answer Selection | ASNQ | ELECTRA-Base + SSP | MRR | 0.757 | #2 of 3 | Archive leaderboard | report |
| Question Answering | TrecQA | DeBERTa-V3-Large + SSP | MAP | 0.923 | #3 of 13 | Archive leaderboard | report |
| Question Answering | TrecQA | DeBERTa-V3-Large + SSP | MRR | 0.946 | #3 of 13 | Archive leaderboard | report |
| Question Answering | TrecQA | RoBERTa-Base + PSD | MAP | 0.903 | #7 of 13 | Archive leaderboard | report |
| Question Answering | TrecQA | RoBERTa-Base + PSD | MRR | 0.951 | #7 of 13 | Archive leaderboard | report |
| Question Answering | WikiQA | DeBERTa-V3-Large + ALL | MAP | 0.909 | #4 of 25 | Archive leaderboard | report |
| Question Answering | WikiQA | DeBERTa-V3-Large + ALL | MRR | 0.920 | #4 of 25 | Archive leaderboard | report |
| Question Answering | WikiQA | DeBERTa-Large + SSP | MAP | 0.901 | #5 of 25 | Archive leaderboard | report |
| Question Answering | WikiQA | DeBERTa-Large + SSP | MRR | 0.914 | #5 of 25 | Archive leaderboard | report |
| Question Answering | WikiQA | RoBERTa-Base + SSP | MAP | 0.887 | #7 of 25 | Archive leaderboard | report |
| Question Answering | WikiQA | RoBERTa-Base + SSP | MRR | 0.899 | #7 of 25 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections