{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kilt-a-benchmark-for-knowledge-intensive","title":"KILT: a Benchmark for Knowledge Intensive Language Tasks","arxiv_id":"2009.02252","date":"2020-09-04","proceeding":"NAACL 2021 4","authors":["Fabio Petroni","Aleksandra Piktus","Angela Fan","Patrick Lewis","Majid Yazdani","Nicola De Cao","James Thorne","Yacine Jernite","Vladimir Karpukhin","Jean Maillard","Vassilis Plachouras","Tim Rocktäschel","Sebastian Riedel"],"abstract":"Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models do well on individual tasks, developing general models is difficult as each task might require computationally expensive indexing of custom knowledge sources, in addition to dedicated infrastructure. To catalyze research on models that condition on specific information in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot of Wikipedia, reducing engineering turnaround through the re-use of components, as well as accelerating research into task-agnostic memory architectures. We test both task-specific and general baselines, evaluating downstream performance in addition to the ability of the models to provide provenance. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline, outperforming more tailor-made approaches for fact checking, open-domain question answering and dialogue, and yielding competitive results on entity linking and slot filling, by generating disambiguated text. KILT data and code are available at https://github.com/facebookresearch/KILT.","url_abs":"https://arxiv.org/abs/2009.02252v4","url_pdf":"https://arxiv.org/pdf/2009.02252v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kilt-a-benchmark-for-knowledge-intensive","repo_url":"https://github.com/facebookresearch/KILT","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"kilt-a-benchmark-for-knowledge-intensive","repo_url":"https://github.com/facebookresearch/editeval","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"CC0-1.0"}},{"paper_slug":"kilt-a-benchmark-for-knowledge-intensive","repo_url":"https://github.com/zouharvi/kb-shrink","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"entity-linking","task_name":"Entity Linking"},{"task_slug":"fact-checking","task_name":"Fact Checking"},{"task_slug":"fact-verification","task_name":"Fact Verification"},{"task_slug":"open-domain-dialog","task_name":"Open-Domain Dialog"},{"task_slug":"open-domain-question-answering","task_name":"Open-Domain Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"slot-filling","task_name":"Slot Filling"}],"methods":[{"method_slug":"lstm","method_name":"LSTM"},{"method_slug":"seq2seq","method_name":"Seq2Seq"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"tanh-activation","method_name":"Tanh Activation"}],"datasets_introduced":[{"slug":"kilt","name":"KILT","full_name":"KILT Benchmark"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/entity-linking-on-kilt-aida-yago2","task":"Entity Linking","dataset":"KILT: AIDA-YAGO2","model":"T5-base","rank_in_archive_order":5,"of":11,"metrics":{"Accuracy":"74.05","KILT-AC":"74.05","R-Prec":"74.05","Recall@5":"74.05"},"uses_additional_data":false},{"leaderboard":"/sota/entity-linking-on-kilt-wned-cweb","task":"Entity Linking","dataset":"KILT: WNED-CWEB","model":"T5-base","rank_in_archive_order":3,"of":10,"metrics":{"Accuracy":"49.29","KILT-AC":"49.29","R-Prec":"49.29","Recall@5":"49.29"},"uses_additional_data":false},{"leaderboard":"/sota/entity-linking-on-kilt-wned-wiki","task":"Entity Linking","dataset":"KILT: WNED-WIKI","model":"T5-base","rank_in_archive_order":4,"of":10,"metrics":{"Accuracy":"47.13","KILT-AC":"47.13","R-Prec":"47.13","Recall@5":"47.13"},"uses_additional_data":false},{"leaderboard":"/sota/fact-verification-on-kilt-fever","task":"Fact Verification","dataset":"KILT: FEVER","model":"RAG","rank_in_archive_order":7,"of":33,"metrics":{"Accuracy":"86.31","KILT-AC":"53.45","R-Prec":"61.94","Recall@5":"75.55"},"uses_additional_data":false},{"leaderboard":"/sota/fact-verification-on-kilt-fever","task":"Fact Verification","dataset":"KILT: FEVER","model":"T5-base","rank_in_archive_order":17,"of":33,"metrics":{"Accuracy":"76.3","KILT-AC":"0.0","R-Prec":"0.0","Recall@5":"0.0"},"uses_additional_data":false},{"leaderboard":"/sota/open-domain-dialog-on-kilt-wizard-of","task":"Open-Domain Dialog","dataset":"KILT: Wizard of Wikipedia","model":"T5-base","rank_in_archive_order":20,"of":21,"metrics":{"F1":"13.53","KILT-F1":"0.0","KILT-RL":"0.0","R-Prec":"0.0","ROUGE-L":"12.4","Recall@5":"0.0"},"uses_additional_data":false},{"leaderboard":"/sota/open-domain-question-answering-on-kilt-eli5","task":"Open-Domain Question Answering","dataset":"KILT: ELI5","model":"T5-base","rank_in_archive_order":11,"of":16,"metrics":{"F1":"16.1","KILT-F1":"0.0","KILT-RL":"0.0","R-Prec":"0.0","ROUGE-L":"19.08","Recall@5":"0.0"},"uses_additional_data":false},{"leaderboard":"/sota/open-domain-question-answering-on-kilt-1","task":"Open-Domain Question Answering","dataset":"KILT: HotpotQA","model":"T5-base","rank_in_archive_order":13,"of":14,"metrics":{"EM":"12.64","F1":"19.57","KILT-EM":"0.0","KILT-F1":"0.0","R-Prec":"0.0","Recall@5":"0.0"},"uses_additional_data":false},{"leaderboard":"/sota/open-domain-question-answering-on-kilt","task":"Open-Domain Question Answering","dataset":"KILT: Natural Questions","model":"T5-base","rank_in_archive_order":15,"of":16,"metrics":{"EM":"19.6","F1":"27.73","KILT-EM":"0.0","KILT-F1":"0.0","R-Prec":"0.0","Recall@5":"0.0"},"uses_additional_data":false},{"leaderboard":"/sota/open-domain-question-answering-on-kilt-2","task":"Open-Domain Question Answering","dataset":"KILT: TriviaQA","model":"T5-base","rank_in_archive_order":15,"of":15,"metrics":{"EM":"18.11","F1":"27.83","KILT-EM":"0.0","KILT-F1":"0.0","R-Prec":"0.0","Recall@5":"0.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-kilt-eli5","task":"Question Answering","dataset":"KILT: ELI5","model":"T5-base","rank_in_archive_order":5,"of":7,"metrics":{"F1":"16.1","Rouge-L":"19.08"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-kilt-eli5","task":"Question Answering","dataset":"KILT: ELI5","model":"BART+DPR","rank_in_archive_order":6,"of":7,"metrics":{"F1":"17.88","Rouge-L":"17.41"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-kilt-eli5","task":"Question Answering","dataset":"KILT: ELI5","model":"RAG","rank_in_archive_order":7,"of":7,"metrics":{"F1":"14.51","Rouge-L":"14.05"},"uses_additional_data":false},{"leaderboard":"/sota/slot-filling-on-kilt-t-rex","task":"Slot Filling","dataset":"KILT: T-REx","model":"T5-base","rank_in_archive_order":18,"of":20,"metrics":{"Accuracy":"43.56","F1":"50.61","KILT-AC":"0.0","KILT-F1":"0.0","R-Prec":"0.0","Recall@5":"0.0"},"uses_additional_data":false},{"leaderboard":"/sota/slot-filling-on-kilt-zero-shot-re","task":"Slot Filling","dataset":"KILT: Zero Shot RE","model":"T5-base","rank_in_archive_order":20,"of":21,"metrics":{"Accuracy":"9.02","F1":"13.52","KILT-AC":"0.0","KILT-F1":"0.0","R-Prec":"0.0","Recall@5":"0.0"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2009.02252","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2009.02252"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zouharvi/kb-shrink","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/KILT","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/editeval","reach":{"status":"ok","spdx":"CC0-1.0"}}],"summary":{"ran_honours":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"d5d38123e280f4e0","entry":"get_train_data","repo":"zouharvi/kb-shrink","repo_kind":"listed","path":"src/reduce_dim/autoencoder/model.py","file_url":"https://github.com/zouharvi/kb-shrink/blob/HEAD/src/reduce_dim/autoencoder/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d5d38123e280f4e0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}