{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/few-shot-question-answering-by-pretraining","title":"Few-Shot Question Answering by Pretraining Span Selection","arxiv_id":"2101.00438","date":"2021-01-02","proceeding":"ACL 2021 5","authors":["Ori Ram","Yuval Kirstain","Jonathan Berant","Amir Globerson","Omer Levy"],"abstract":"In several question answering benchmarks, pretrained models have reached human parity through fine-tuning on an order of 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred training examples are available, and observe that standard models perform poorly, highlighting the discrepancy between current pretraining objectives and question answering. We propose a new pretraining scheme tailored for question answering: recurring span selection. Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans are replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select the answer span. The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD with only 128 training examples), while maintaining competitive performance in the high-resource setting.","url_abs":"https://arxiv.org/abs/2101.00438v2","url_pdf":"https://arxiv.org/pdf/2101.00438v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"few-shot-question-answering-by-pretraining","repo_url":"https://github.com/oriram/splinter","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"unanswered"}},{"paper_slug":"few-shot-question-answering-by-pretraining","repo_url":"https://github.com/ednussi/thesis_public","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"few-shot-question-answering-by-pretraining","repo_url":"https://github.com/ncpaddle/splinter-paddlepaddle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"paddle","reach":{"status":"unanswered"}},{"paper_slug":"few-shot-question-answering-by-pretraining","repo_url":"https://github.com/zhoucz97/Splinter-paddle","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"paddle","reach":null},{"paper_slug":"few-shot-question-answering-by-pretraining","repo_url":"https://github.com/yangyucheng000/University/tree/main/model-3/splinter","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2101.00438","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2101.00438"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ednussi/thesis_public","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ncpaddle/splinter-paddlepaddle","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhoucz97/Splinter-paddle","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangyucheng000/University/tree/main/model-3/splinter","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/oriram/splinter","reach":{"status":"unanswered"}}],"summary":{"ran":2,"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"fd274e8bb8c2256f","entry":"FullyConnectedLayer","repo":"zhoucz97/Splinter-paddle","repo_kind":"listed","path":"finetuning/pytorch_modeling.py","file_url":"https://github.com/zhoucz97/Splinter-paddle/blob/HEAD/finetuning/pytorch_modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd274e8bb8c2256f"}},{"code_sha256_prefix":"1a960ffb78787c07","entry":"QuestionAwareSpanSelectionHead","repo":"zhoucz97/Splinter-paddle","repo_kind":"listed","path":"finetuning/pytorch_modeling.py","file_url":"https://github.com/zhoucz97/Splinter-paddle/blob/HEAD/finetuning/pytorch_modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1a960ffb78787c07"}},{"code_sha256_prefix":"3c52fe0bb5132f74","entry":"gather_positions","repo":"zhoucz97/Splinter-paddle","repo_kind":"listed","path":"finetuning/pytorch_modeling.py","file_url":"https://github.com/zhoucz97/Splinter-paddle/blob/HEAD/finetuning/pytorch_modeling.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c52fe0bb5132f74"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}