{"url":"/sota/question-answering-on-squad11-dev","task":{"name":"Question Answering","url":"/task/question-answering","note":null},"dataset":{"name":"SQuAD1.1 dev","url":"/dataset/squad"},"category":"Natural Language Processing","categories":["Miscellaneous","Natural Language Processing","Reasoning"],"category_note":null,"description":"Question answering can be segmented into domain-specific tasks like community question answering and knowledge-base question answering. Popular benchmark datasets for evaluation question answering systems include [SQuAD](/dataset/squad), [HotPotQA](/dataset/hotpotqa), [bAbI](/dataset/babi-1), [TriviaQA](/dataset/triviaqa), [WikiQA](/dataset/wikiqa), and many others. Models for question answering are typically evaluated on metrics like EM and F1. Some recent top performing models are T5 and XLNet.\r\n\r\n( Image credit: [SQuAD](https://rajpurkar.github.io/mlx/qa-and-squad/) )","description_from":"task","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","rank":"the archive's row order at snapshot; not re-ranked","rows_end_at":"2025-07-28","rows_withheld_as_spam":0,"metric_values":"the archive's strings, untouched"},"metrics":["EM","F1"],"metric_direction":{"note":"inferred from the metric name only (the archive records no direction); null = not inferred, chart draws points only","by_metric":{"EM":null,"F1":"higher"}},"counts":{"rows":55,"rows_with_code":45,"rows_with_paper_page":55,"rows_dated":55,"rows_using_additional_data":7},"rows":[{"rank_in_archive_order":1,"model":"T5-11B","metrics":{"EM":"90.06","F1":"95.64"},"uses_additional_data":true,"paper_date":"2019-10-23","paper":"/paper/exploring-the-limits-of-transfer-learning","paper_url":"https://arxiv.org/abs/1910.10683v4","paper_title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":2,"n_unverified":29,"n_samples":31,"n_pointer_only_licence":0}},{"rank_in_archive_order":2,"model":"LUKE","metrics":{"EM":"89.8"},"uses_additional_data":true,"paper_date":"2020-10-02","paper":"/paper/luke-deep-contextualized-entity","paper_url":"https://arxiv.org/abs/2010.01057v1","paper_title":"LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention","code":"https://github.com/huggingface/transformers","n_code_links":9,"syntology":{"n_ran":3,"n_unverified":7,"n_samples":10,"n_pointer_only_licence":0}},{"rank_in_archive_order":3,"model":"XLNet+DSC","metrics":{"EM":"89.79","F1":"95.77"},"uses_additional_data":true,"paper_date":"2019-11-07","paper":"/paper/dice-loss-for-data-imbalanced-nlp-tasks","paper_url":"https://arxiv.org/abs/1911.02855v3","paper_title":"Dice Loss for Data-imbalanced NLP Tasks","code":"https://github.com/ShannonAI/dice_loss_for_NLP","n_code_links":4,"syntology":{"n_ran":0,"n_unverified":3,"n_samples":3,"n_pointer_only_licence":0}},{"rank_in_archive_order":4,"model":"XLNet (single model)","metrics":{"EM":"89.7","F1":"95.1"},"uses_additional_data":true,"paper_date":"2019-06-19","paper":"/paper/xlnet-generalized-autoregressive-pretraining","paper_url":"https://arxiv.org/abs/1906.08237v2","paper_title":"XLNet: Generalized Autoregressive Pretraining for Language Understanding","code":"https://github.com/huggingface/transformers","n_code_links":27,"syntology":{"n_ran":10,"n_unverified":14,"n_samples":24,"n_pointer_only_licence":3}},{"rank_in_archive_order":5,"model":"T5-3B","metrics":{"EM":"88.53","F1":"94.95"},"uses_additional_data":true,"paper_date":"2019-10-23","paper":"/paper/exploring-the-limits-of-transfer-learning","paper_url":"https://arxiv.org/abs/1910.10683v4","paper_title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":2,"n_unverified":29,"n_samples":31,"n_pointer_only_licence":0}},{"rank_in_archive_order":6,"model":"T5-Large 770M","metrics":{"EM":"86.66","F1":"93.79"},"uses_additional_data":false,"paper_date":"2019-10-23","paper":"/paper/exploring-the-limits-of-transfer-learning","paper_url":"https://arxiv.org/abs/1910.10683v4","paper_title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":2,"n_unverified":29,"n_samples":31,"n_pointer_only_licence":0}},{"rank_in_archive_order":7,"model":"BERT-LARGE (Ensemble+TriviaQA)","metrics":{"EM":"86.2","F1":"92.2"},"uses_additional_data":false,"paper_date":"2018-10-11","paper":"/paper/bert-pre-training-of-deep-bidirectional","paper_url":"https://arxiv.org/abs/1810.04805v2","paper_title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","code":"https://github.com/huggingface/transformers","n_code_links":534,"syntology":{"n_ran":204,"n_unverified":455,"n_samples":659,"n_pointer_only_licence":149}},{"rank_in_archive_order":8,"model":"T5-Base","metrics":{"EM":"85.44","F1":"92.08"},"uses_additional_data":true,"paper_date":"2019-10-23","paper":"/paper/exploring-the-limits-of-transfer-learning","paper_url":"https://arxiv.org/abs/1910.10683v4","paper_title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":2,"n_unverified":29,"n_samples":31,"n_pointer_only_licence":0}},{"rank_in_archive_order":9,"model":"BERT-LARGE (Single+TriviaQA)","metrics":{"EM":"84.2","F1":"91.1"},"uses_additional_data":false,"paper_date":"2018-10-11","paper":"/paper/bert-pre-training-of-deep-bidirectional","paper_url":"https://arxiv.org/abs/1810.04805v2","paper_title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","code":"https://github.com/huggingface/transformers","n_code_links":534,"syntology":{"n_ran":204,"n_unverified":455,"n_samples":659,"n_pointer_only_licence":149}},{"rank_in_archive_order":10,"model":"BERT-Large-uncased-PruneOFA (90% unstruct sparse)","metrics":{"EM":"83.35","F1":"90.2"},"uses_additional_data":false,"paper_date":"2021-11-10","paper":"/paper/prune-once-for-all-sparse-pre-trained","paper_url":"https://arxiv.org/abs/2111.05754v1","paper_title":"Prune Once for All: Sparse Pre-Trained Language Models","code":"https://github.com/intel/intel-extension-for-transformers","n_code_links":2,"syntology":null},{"rank_in_archive_order":11,"model":"BERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8)","metrics":{"EM":"83.22","F1":"90.02"},"uses_additional_data":false,"paper_date":"2021-11-10","paper":"/paper/prune-once-for-all-sparse-pre-trained","paper_url":"https://arxiv.org/abs/2111.05754v1","paper_title":"Prune Once for All: Sparse Pre-Trained Language Models","code":"https://github.com/intel/intel-extension-for-transformers","n_code_links":2,"syntology":null},{"rank_in_archive_order":12,"model":"BERT-Base-uncased-PruneOFA (85% unstruct sparse)","metrics":{"EM":"81.1","F1":"88.42"},"uses_additional_data":false,"paper_date":"2021-11-10","paper":"/paper/prune-once-for-all-sparse-pre-trained","paper_url":"https://arxiv.org/abs/2111.05754v1","paper_title":"Prune Once for All: Sparse Pre-Trained Language Models","code":"https://github.com/intel/intel-extension-for-transformers","n_code_links":2,"syntology":null},{"rank_in_archive_order":13,"model":"BERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8)","metrics":{"EM":"80.84","F1":"88.24"},"uses_additional_data":false,"paper_date":"2021-11-10","paper":"/paper/prune-once-for-all-sparse-pre-trained","paper_url":"https://arxiv.org/abs/2111.05754v1","paper_title":"Prune Once for All: Sparse Pre-Trained Language Models","code":"https://github.com/intel/intel-extension-for-transformers","n_code_links":2,"syntology":null},{"rank_in_archive_order":14,"model":"BERT-Base-uncased-PruneOFA (90% unstruct sparse)","metrics":{"EM":"79.83","F1":"87.25"},"uses_additional_data":false,"paper_date":"2021-11-10","paper":"/paper/prune-once-for-all-sparse-pre-trained","paper_url":"https://arxiv.org/abs/2111.05754v1","paper_title":"Prune Once for All: Sparse Pre-Trained Language Models","code":"https://github.com/intel/intel-extension-for-transformers","n_code_links":2,"syntology":null},{"rank_in_archive_order":15,"model":"TinyBERT-6 67M","metrics":{"EM":"79.7","F1":"87.5"},"uses_additional_data":false,"paper_date":"2019-09-23","paper":"/paper/190910351","paper_url":"https://arxiv.org/abs/1909.10351v5","paper_title":"TinyBERT: Distilling BERT for Natural Language Understanding","code":"https://github.com/PaddlePaddle/PaddleNLP/tree/develop/paddlenlp/transformers/tinybert","n_code_links":10,"syntology":{"n_ran":0,"n_unverified":4,"n_samples":4,"n_pointer_only_licence":4}},{"rank_in_archive_order":16,"model":"T5-Small","metrics":{"EM":"79.1","F1":"87.24"},"uses_additional_data":true,"paper_date":"2019-10-23","paper":"/paper/exploring-the-limits-of-transfer-learning","paper_url":"https://arxiv.org/abs/1910.10683v4","paper_title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":2,"n_unverified":29,"n_samples":31,"n_pointer_only_licence":0}},{"rank_in_archive_order":17,"model":"R.M-Reader (single)","metrics":{"EM":"78.9","F1":" 86.3"},"uses_additional_data":false,"paper_date":"2017-05-08","paper":"/paper/reinforced-mnemonic-reader-for-machine","paper_url":"http://arxiv.org/abs/1705.02798v6","paper_title":"Reinforced Mnemonic Reader for Machine Reading Comprehension","code":"https://github.com/HKUST-KnowComp/MnemonicReader","n_code_links":3,"syntology":null},{"rank_in_archive_order":18,"model":"DensePhrases","metrics":{"EM":"78.3","F1":"86.3"},"uses_additional_data":false,"paper_date":"2020-12-23","paper":"/paper/learning-dense-representations-of-phrases-at","paper_url":"https://arxiv.org/abs/2012.12624v3","paper_title":"Learning Dense Representations of Phrases at Scale","code":"https://github.com/princeton-nlp/SimCSE","n_code_links":4,"syntology":null},{"rank_in_archive_order":19,"model":"DistilBERT-uncased-PruneOFA (85% unstruct sparse)","metrics":{"EM":"78.1","F1":"85.82"},"uses_additional_data":false,"paper_date":"2021-11-10","paper":"/paper/prune-once-for-all-sparse-pre-trained","paper_url":"https://arxiv.org/abs/2111.05754v1","paper_title":"Prune Once for All: Sparse Pre-Trained Language Models","code":"https://github.com/intel/intel-extension-for-transformers","n_code_links":2,"syntology":null},{"rank_in_archive_order":20,"model":"DistilBERT","metrics":{"EM":"77.7"},"uses_additional_data":false,"paper_date":"2019-10-02","paper":"/paper/distilbert-a-distilled-version-of-bert","paper_url":"https://arxiv.org/abs/1910.01108v4","paper_title":"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter","code":"https://github.com/huggingface/transformers","n_code_links":37,"syntology":{"n_ran":19,"n_unverified":8,"n_samples":27,"n_pointer_only_licence":0}},{"rank_in_archive_order":21,"model":"DistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8)","metrics":{"EM":"77.03","F1":"85.13"},"uses_additional_data":false,"paper_date":"2021-11-10","paper":"/paper/prune-once-for-all-sparse-pre-trained","paper_url":"https://arxiv.org/abs/2111.05754v1","paper_title":"Prune Once for All: Sparse Pre-Trained Language Models","code":"https://github.com/intel/intel-extension-for-transformers","n_code_links":2,"syntology":null},{"rank_in_archive_order":22,"model":"DistilBERT-uncased-PruneOFA (90% unstruct sparse)","metrics":{"EM":"76.91","F1":"84.82"},"uses_additional_data":false,"paper_date":"2021-11-10","paper":"/paper/prune-once-for-all-sparse-pre-trained","paper_url":"https://arxiv.org/abs/2111.05754v1","paper_title":"Prune Once for All: Sparse Pre-Trained Language Models","code":"https://github.com/intel/intel-extension-for-transformers","n_code_links":2,"syntology":null},{"rank_in_archive_order":23,"model":"KAR","metrics":{"EM":"76.7","F1":"84.9 "},"uses_additional_data":false,"paper_date":"2018-09-10","paper":"/paper/exploring-machine-reading-comprehension-with","paper_url":"https://arxiv.org/abs/1809.03449v3","paper_title":"Explicit Utilization of General Knowledge in Machine Reading Comprehension","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":24,"model":"SAN (single)","metrics":{"EM":"76.235","F1":"84.056"},"uses_additional_data":false,"paper_date":"2017-12-10","paper":"/paper/stochastic-answer-networks-for-machine","paper_url":"http://arxiv.org/abs/1712.03556v2","paper_title":"Stochastic Answer Networks for Machine Reading Comprehension","code":"https://github.com/kevinduh/san_mrc","n_code_links":6,"syntology":null},{"rank_in_archive_order":25,"model":"DistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8)","metrics":{"EM":"75.62","F1":"83.87"},"uses_additional_data":false,"paper_date":"2021-11-10","paper":"/paper/prune-once-for-all-sparse-pre-trained","paper_url":"https://arxiv.org/abs/2111.05754v1","paper_title":"Prune Once for All: Sparse Pre-Trained Language Models","code":"https://github.com/intel/intel-extension-for-transformers","n_code_links":2,"syntology":null},{"rank_in_archive_order":26,"model":"FusionNet","metrics":{"EM":"75.3","F1":"83.6"},"uses_additional_data":false,"paper_date":"2017-11-16","paper":"/paper/fusionnet-fusing-via-fully-aware-attention","paper_url":"http://arxiv.org/abs/1711.07341v2","paper_title":"FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension","code":"https://github.com/momohuang/FusionNet-NLI","n_code_links":3,"syntology":null},{"rank_in_archive_order":27,"model":"QANet (data aug x3)","metrics":{"EM":"75.1","F1":"83.8"},"uses_additional_data":false,"paper_date":"2018-04-23","paper":"/paper/qanet-combining-local-convolution-with-global","paper_url":"http://arxiv.org/abs/1804.09541v1","paper_title":"QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension","code":"https://github.com/BangLiu/QANet-PyTorch","n_code_links":15,"syntology":{"n_ran":7,"n_unverified":12,"n_samples":19,"n_pointer_only_licence":2}},{"rank_in_archive_order":28,"model":"QANet (data aug x2)","metrics":{"EM":"74.5","F1":"83.2"},"uses_additional_data":false,"paper_date":"2018-04-23","paper":"/paper/qanet-combining-local-convolution-with-global","paper_url":"http://arxiv.org/abs/1804.09541v1","paper_title":"QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension","code":"https://github.com/BangLiu/QANet-PyTorch","n_code_links":15,"syntology":{"n_ran":7,"n_unverified":12,"n_samples":19,"n_pointer_only_licence":2}},{"rank_in_archive_order":29,"model":"DCN+ (single)","metrics":{"EM":"74.5","F1":"83.1"},"uses_additional_data":false,"paper_date":"2017-10-31","paper":"/paper/dcn-mixed-objective-and-deep-residual","paper_url":"http://arxiv.org/abs/1711.00106v2","paper_title":"DCN+: Mixed Objective and Deep Residual Coattention for Question Answering","code":"https://github.com/lmn-extracts/dcn_plus","n_code_links":1,"syntology":null},{"rank_in_archive_order":30,"model":"QANet","metrics":{"EM":"73.6","F1":"82.7"},"uses_additional_data":false,"paper_date":"2018-04-23","paper":"/paper/qanet-combining-local-convolution-with-global","paper_url":"http://arxiv.org/abs/1804.09541v1","paper_title":"QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension","code":"https://github.com/BangLiu/QANet-PyTorch","n_code_links":15,"syntology":{"n_ran":7,"n_unverified":12,"n_samples":19,"n_pointer_only_licence":2}},{"rank_in_archive_order":31,"model":"PhaseCond (single)","metrics":{"EM":"72.1","F1":"81.4"},"uses_additional_data":false,"paper_date":"2017-10-28","paper":"/paper/phase-conductor-on-multi-layered-attentions","paper_url":"http://arxiv.org/abs/1710.10504v2","paper_title":"Phase Conductor on Multi-layered Attentions for Machine Comprehension","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":32,"model":"SRU","metrics":{"EM":"71.4","F1":"80.2"},"uses_additional_data":false,"paper_date":"2017-09-08","paper":"/paper/simple-recurrent-units-for-highly","paper_url":"http://arxiv.org/abs/1709.02755v5","paper_title":"Simple Recurrent Units for Highly Parallelizable Recurrence","code":"https://github.com/aymericdamien/TopDeepLearning","n_code_links":11,"syntology":{"n_ran":1,"n_unverified":0,"n_samples":1,"n_pointer_only_licence":0}},{"rank_in_archive_order":33,"model":"Smarnet","metrics":{"EM":"71.362","F1":"80.183"},"uses_additional_data":false,"paper_date":"2017-10-08","paper":"/paper/smarnet-teaching-machines-to-read-and","paper_url":"http://arxiv.org/abs/1710.02772v1","paper_title":"Smarnet: Teaching Machines to Read and Comprehend Like Human","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":34,"model":"DCN (Char + CoVe)","metrics":{"EM":"71.3","F1":"79.9"},"uses_additional_data":false,"paper_date":"2017-08-01","paper":"/paper/learned-in-translation-contextualized-word","paper_url":"http://arxiv.org/abs/1708.00107v2","paper_title":"Learned in Translation: Contextualized Word Vectors","code":"https://github.com/salesforce/cove","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":2}},{"rank_in_archive_order":35,"model":"R-NET (single)","metrics":{"EM":"71.1","F1":"79.5"},"uses_additional_data":false,"paper_date":"2017-07-01","paper":"/paper/gated-self-matching-networks-for-reading","paper_url":"https://aclanthology.org/P17-1018","paper_title":"Gated Self-Matching Networks for Reading Comprehension and Question Answering","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":36,"model":"Ruminating Reader","metrics":{"EM":"70.6","F1":"79.5"},"uses_additional_data":false,"paper_date":"2017-04-24","paper":"/paper/ruminating-reader-reasoning-with-gated-multi","paper_url":"http://arxiv.org/abs/1704.07415v1","paper_title":"Ruminating Reader: Reasoning with Gated Multi-Hop Attention","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":37,"model":"FastQAExt (beam-size 5)","metrics":{"EM":"70.3","F1":"78.5"},"uses_additional_data":false,"paper_date":"2017-03-14","paper":"/paper/making-neural-qa-as-simple-as-possible-but","paper_url":"http://arxiv.org/abs/1703.04816v3","paper_title":"Making Neural QA as Simple as Possible but not Simpler","code":"https://github.com/uclmr/jack","n_code_links":3,"syntology":{"n_ran":0,"n_unverified":2,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":38,"model":"DrQA (Document Reader only)","metrics":{"EM":"69.5","F1":"78.8"},"uses_additional_data":false,"paper_date":"2017-03-31","paper":"/paper/reading-wikipedia-to-answer-open-domain","paper_url":"http://arxiv.org/abs/1704.00051v2","paper_title":"Reading Wikipedia to Answer Open-Domain Questions","code":"https://github.com/facebookresearch/ParlAI","n_code_links":10,"syntology":{"n_ran":1,"n_unverified":0,"n_samples":1,"n_pointer_only_licence":1}},{"rank_in_archive_order":39,"model":"jNet (TreeLSTM adaptation, QTLa, K=100)","metrics":{"EM":"69.10","F1":"78.38"},"uses_additional_data":false,"paper_date":"2017-03-14","paper":"/paper/exploring-question-understanding-and","paper_url":"http://arxiv.org/abs/1703.04617v2","paper_title":"Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":40,"model":"SEDT-LSTM","metrics":{"EM":" 67.89","F1":" 77.42 "},"uses_additional_data":false,"paper_date":"2017-03-02","paper":"/paper/structural-embedding-of-syntactic-trees-for","paper_url":"http://arxiv.org/abs/1703.00572v3","paper_title":"Structural Embedding of Syntactic Trees for Machine Comprehension","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":41,"model":"BIDAF (single)","metrics":{"EM":" 67.7","F1":"77.3"},"uses_additional_data":false,"paper_date":"2016-11-05","paper":"/paper/bidirectional-attention-flow-for-machine","paper_url":"http://arxiv.org/abs/1611.01603v6","paper_title":"Bidirectional Attention Flow for Machine Comprehension","code":"https://github.com/allenai/bi-att-flow","n_code_links":27,"syntology":{"n_ran":8,"n_unverified":3,"n_samples":11,"n_pointer_only_licence":7}},{"rank_in_archive_order":42,"model":"SECT-LSTM","metrics":{"EM":"67.65","F1":"77.19"},"uses_additional_data":false,"paper_date":"2017-03-02","paper":"/paper/structural-embedding-of-syntactic-trees-for","paper_url":"http://arxiv.org/abs/1703.00572v3","paper_title":"Structural Embedding of Syntactic Trees for Machine Comprehension","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":43,"model":"RASOR","metrics":{"EM":"66.4","F1":"74.9"},"uses_additional_data":false,"paper_date":"2016-11-04","paper":"/paper/learning-recurrent-span-representations-for","paper_url":"http://arxiv.org/abs/1611.01436v2","paper_title":"Learning Recurrent Span Representations for Extractive Question Answering","code":"https://github.com/shimisalant/RaSoR","n_code_links":2,"syntology":null},{"rank_in_archive_order":44,"model":"MPCM","metrics":{"EM":"66.1","F1":"75.8"},"uses_additional_data":false,"paper_date":"2016-12-13","paper":"/paper/multi-perspective-context-matching-for","paper_url":"http://arxiv.org/abs/1612.04211v1","paper_title":"Multi-Perspective Context Matching for Machine Comprehension","code":"https://github.com/bloomsburyai/question-generation","n_code_links":1,"syntology":null},{"rank_in_archive_order":45,"model":"DCN","metrics":{"EM":"65.4","F1":"75.6"},"uses_additional_data":false,"paper_date":"2016-11-05","paper":"/paper/dynamic-coattention-networks-for-question","paper_url":"http://arxiv.org/abs/1611.01604v4","paper_title":"Dynamic Coattention Networks For Question Answering","code":"https://github.com/lmn-extracts/dcn_plus","n_code_links":6,"syntology":null},{"rank_in_archive_order":46,"model":"FABIR","metrics":{"EM":"65.1","F1":"75.6"},"uses_additional_data":false,"paper_date":"2018-10-22","paper":"/paper/a-fully-attention-based-information-retriever","paper_url":"http://arxiv.org/abs/1810.09580v1","paper_title":"A Fully Attention-Based Information Retriever","code":"https://github.com/AlCorreia/FABIR","n_code_links":1,"syntology":null},{"rank_in_archive_order":47,"model":"Match-LSTM with Bi-Ans-Ptr (Boundary+Search+b)","metrics":{"EM":"64.1","F1":" 64.7"},"uses_additional_data":false,"paper_date":"2016-08-29","paper":"/paper/machine-comprehension-using-match-lstm-and","paper_url":"http://arxiv.org/abs/1608.07905v2","paper_title":"Machine Comprehension Using Match-LSTM and Answer Pointer","code":"https://github.com/baidu/DuReader","n_code_links":5,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":3}},{"rank_in_archive_order":48,"model":"OTF dict+spelling (single)","metrics":{"EM":"63.06"},"uses_additional_data":false,"paper_date":"2017-06-01","paper":"/paper/learning-to-compute-word-embeddings-on-the","paper_url":"http://arxiv.org/abs/1706.00286v3","paper_title":"Learning to Compute Word Embeddings On the Fly","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":49,"model":"DCR","metrics":{"EM":" 62.5","F1":"71.2"},"uses_additional_data":false,"paper_date":"2016-10-31","paper":"/paper/end-to-end-answer-chunk-extraction-and","paper_url":"http://arxiv.org/abs/1610.09996v2","paper_title":"End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":50,"model":"FG fine-grained gate","metrics":{"EM":"59.95","F1":"71.25"},"uses_additional_data":false,"paper_date":"2016-11-06","paper":"/paper/words-or-characters-fine-grained-gating-for","paper_url":"http://arxiv.org/abs/1611.01724v2","paper_title":"Words or Characters? Fine-grained Gating for Reading Comprehension","code":"https://github.com/kimiyoung/fg-gating","n_code_links":1,"syntology":null},{"rank_in_archive_order":51,"model":"LUKE 483M","metrics":{"F1":"95"},"uses_additional_data":false,"paper_date":"2020-10-02","paper":"/paper/luke-deep-contextualized-entity","paper_url":"https://arxiv.org/abs/2010.01057v1","paper_title":"LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention","code":"https://github.com/huggingface/transformers","n_code_links":9,"syntology":{"n_ran":3,"n_unverified":7,"n_samples":10,"n_pointer_only_licence":0}},{"rank_in_archive_order":52,"model":"BART Base (with text infilling)","metrics":{"F1":"90.8"},"uses_additional_data":false,"paper_date":"2019-10-29","paper":"/paper/bart-denoising-sequence-to-sequence-pre","paper_url":"https://arxiv.org/abs/1910.13461v1","paper_title":"BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension","code":"https://github.com/huggingface/transformers","n_code_links":47,"syntology":{"n_ran":22,"n_unverified":31,"n_samples":53,"n_pointer_only_licence":7}},{"rank_in_archive_order":53,"model":"BERT large (LAMB optimizer)","metrics":{"F1":"90.584"},"uses_additional_data":false,"paper_date":"2019-04-01","paper":"/paper/reducing-bert-pre-training-time-from-3-days","paper_url":"https://arxiv.org/abs/1904.00962v5","paper_title":"Large Batch Optimization for Deep Learning: Training BERT in 76 minutes","code":"https://github.com/huggingface/pytorch-image-models","n_code_links":32,"syntology":{"n_ran":1,"n_unverified":10,"n_samples":11,"n_pointer_only_licence":9}},{"rank_in_archive_order":54,"model":"DistilBERT 66M","metrics":{"F1":"85.8"},"uses_additional_data":false,"paper_date":"2019-10-02","paper":"/paper/distilbert-a-distilled-version-of-bert","paper_url":"https://arxiv.org/abs/1910.01108v4","paper_title":"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter","code":"https://github.com/huggingface/transformers","n_code_links":37,"syntology":{"n_ran":19,"n_unverified":8,"n_samples":27,"n_pointer_only_licence":0}},{"rank_in_archive_order":55,"model":"BiDAF + Self Attention + ELMo","metrics":{"F1":"85.6"},"uses_additional_data":false,"paper_date":"2018-02-15","paper":"/paper/deep-contextualized-word-representations","paper_url":"http://arxiv.org/abs/1802.05365v2","paper_title":"Deep contextualized word representations","code":"https://github.com/flairNLP/flair","n_code_links":46,"syntology":{"n_ran":23,"n_unverified":35,"n_samples":58,"n_pointer_only_licence":25}}],"since_archive":{"claim":"Results that newer papers report for their own method, placed here by Syntology. A model pointed at the cell in the paper's own table; the number was read from that cell and checked against this leaderboard's metric, dataset, split and scale; an independent check that saw this leaderboard's other rows and every other leaderboard on the same dataset accepted it. Not reviewed by the paper's authors or by the archive's editors, and not ranked against the archive rows.","extraction_file_present":true,"measurement":{"test_papers":883,"papers_with_output":881,"judged_true":108,"judged":110,"wilson95_lower":0.9361,"measured_on":"2026-09-24","frozen_commit":"0e3de0df94"},"measurement_note":"blind adjudication of accepted entries on a held-out split of archive papers, rules frozen before the test","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","complete":false,"papers_newer_than_archive":9581,"papers_checked":6264,"papers_extracted_not_yet_verified":0,"boards_without_verdict":2,"papers_not_yet_extracted":3316},"order":"newest first by month (arXiv date, else the arXiv-id month), then arXiv id descending","columns":[],"entries":[]},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per row: N of M harvested code samples from that row's paper executed on a synthesized fixture; the other M-N are unverified. Not a reproduction of the row's number; not a correctness claim. n_pointer_only_licence counts samples the site points at rather than redistributes (a licence axis, independent of ran/unverified).","rows_with_graph_line":26,"rows_with_any_sample_ran":23,"distinct_papers_with_graph_line":17,"distinct_papers_with_any_sample_ran":14,"samples_over_distinct_papers":{"n_ran":307,"n_unverified":613,"n_samples":920,"n_pointer_only_licence":212,"note":"each paper (arXiv id) counted once, however many rows it is behind; this is the page-level figure"},"samples_row_weighted":{"n_ran":555,"n_unverified":1223,"n_samples":1778,"n_pointer_only_licence":365,"note":"row-weighted: a paper behind several rows is counted once per row; inflated relative to samples_over_distinct_papers by design, kept for readers summing the per-row syntology blocks"}}}