{"url":"/sota/coreference-resolution-on-winograd-schema","task":{"name":"Coreference Resolution","url":"/task/coreference-resolution","note":null},"dataset":{"name":"Winograd Schema Challenge","url":"/dataset/wsc"},"category":"Natural Language Processing","categories":["Natural Language Processing"],"category_note":null,"description":"Coreference resolution is the task of clustering mentions in text that refer to the same underlying real world entities.\n\nExample:\n\n```\n               +-----------+\n               |           |\nI voted for Obama because he was most aligned with my values\", she said.\n |                                                 |            |\n +-------------------------------------------------+------------+\n```\n\n\"I\", \"my\", and \"she\" belong to the same cluster and \"Obama\" and \"he\" belong to the same cluster.","description_from":"task","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","rank":"the archive's row order at snapshot; not re-ranked","rows_end_at":"2025-07-28","rows_withheld_as_spam":0,"metric_values":"the archive's strings, untouched"},"metrics":["Accuracy"],"metric_direction":{"note":"inferred from the metric name only (the archive records no direction); null = not inferred, chart draws points only","by_metric":{"Accuracy":"higher"}},"counts":{"rows":82,"rows_with_code":64,"rows_with_paper_page":82,"rows_dated":82,"rows_using_additional_data":0},"rows":[{"rank_in_archive_order":1,"model":"PaLM 540B (fine-tuned)","metrics":{"Accuracy":"100"},"uses_additional_data":false,"paper_date":"2022-04-05","paper":"/paper/palm-scaling-language-modeling-with-pathways-1","paper_url":"https://arxiv.org/abs/2204.02311v5","paper_title":"PaLM: Scaling Language Modeling with Pathways","code":"https://github.com/lucidrains/CoCa-pytorch","n_code_links":7,"syntology":{"n_ran":30,"n_unverified":7,"n_samples":37,"n_pointer_only_licence":0}},{"rank_in_archive_order":2,"model":"Vega v2 6B (KD-based prompt transfer)","metrics":{"Accuracy":"98.6"},"uses_additional_data":false,"paper_date":"2022-12-04","paper":"/paper/toward-efficient-language-model-pretraining","paper_url":"https://arxiv.org/abs/2212.01853v1","paper_title":"Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":3,"model":"UL2 20B (fine-tuned)","metrics":{"Accuracy":"98.1"},"uses_additional_data":false,"paper_date":"2022-05-10","paper":"/paper/unifying-language-learning-paradigms","paper_url":"https://arxiv.org/abs/2205.05131v3","paper_title":"UL2: Unifying Language Learning Paradigms","code":"https://github.com/google-research/google-research","n_code_links":2,"syntology":{"n_ran":0,"n_unverified":16,"n_samples":16,"n_pointer_only_licence":0}},{"rank_in_archive_order":4,"model":"Turing NLR v5 XXL 5.4B (fine-tuned)","metrics":{"Accuracy":"97.3"},"uses_additional_data":false,"paper_date":"2022-12-04","paper":"/paper/toward-efficient-language-model-pretraining","paper_url":"https://arxiv.org/abs/2212.01853v1","paper_title":"Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":5,"model":"ST-MoE-32B 269B (fine-tuned)","metrics":{"Accuracy":"96.6"},"uses_additional_data":false,"paper_date":"2022-02-17","paper":"/paper/designing-effective-sparse-expert-models","paper_url":"https://arxiv.org/abs/2202.08906v2","paper_title":"ST-MoE: Designing Stable and Transferable Sparse Expert Models","code":"https://github.com/tensorflow/mesh","n_code_links":3,"syntology":{"n_ran":5,"n_unverified":0,"n_samples":5,"n_pointer_only_licence":5}},{"rank_in_archive_order":6,"model":"DeBERTa-1.5B","metrics":{"Accuracy":"95.9"},"uses_additional_data":false,"paper_date":"2020-06-05","paper":"/paper/deberta-decoding-enhanced-bert-with","paper_url":"https://arxiv.org/abs/2006.03654v6","paper_title":"DeBERTa: Decoding-enhanced BERT with Disentangled Attention","code":"https://github.com/huggingface/transformers","n_code_links":14,"syntology":{"n_ran":4,"n_unverified":9,"n_samples":13,"n_pointer_only_licence":3}},{"rank_in_archive_order":7,"model":"T5-XXL 11B (fine-tuned)","metrics":{"Accuracy":"93.8"},"uses_additional_data":false,"paper_date":"2019-10-23","paper":"/paper/exploring-the-limits-of-transfer-learning","paper_url":"https://arxiv.org/abs/1910.10683v4","paper_title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","code":"https://github.com/huggingface/transformers","n_code_links":57,"syntology":{"n_ran":2,"n_unverified":29,"n_samples":31,"n_pointer_only_licence":0}},{"rank_in_archive_order":8,"model":"ST-MoE-L 4.1B (fine-tuned)","metrics":{"Accuracy":"93.3"},"uses_additional_data":false,"paper_date":"2022-02-17","paper":"/paper/designing-effective-sparse-expert-models","paper_url":"https://arxiv.org/abs/2202.08906v2","paper_title":"ST-MoE: Designing Stable and Transferable Sparse Expert Models","code":"https://github.com/tensorflow/mesh","n_code_links":3,"syntology":{"n_ran":5,"n_unverified":0,"n_samples":5,"n_pointer_only_licence":5}},{"rank_in_archive_order":9,"model":"RoBERTa-WinoGrande 355M","metrics":{"Accuracy":"90.1"},"uses_additional_data":false,"paper_date":"2019-07-24","paper":"/paper/winogrande-an-adversarial-winograd-schema","paper_url":"https://arxiv.org/abs/1907.10641v2","paper_title":"WinoGrande: An Adversarial Winograd Schema Challenge at Scale","code":"https://github.com/vered1986/self_talk","n_code_links":10,"syntology":null},{"rank_in_archive_order":10,"model":"Flan-T5 XXL (zero -shot)","metrics":{"Accuracy":"89.82"},"uses_additional_data":false,"paper_date":"2022-10-20","paper":"/paper/scaling-instruction-finetuned-language-models","paper_url":"https://arxiv.org/abs/2210.11416v5","paper_title":"Scaling Instruction-Finetuned Language Models","code":"https://github.com/google-research/flan","n_code_links":9,"syntology":{"n_ran":8,"n_unverified":9,"n_samples":17,"n_pointer_only_licence":2}},{"rank_in_archive_order":11,"model":"PaLM 540B (5-shot)","metrics":{"Accuracy":"89.5"},"uses_additional_data":false,"paper_date":"2022-04-05","paper":"/paper/palm-scaling-language-modeling-with-pathways-1","paper_url":"https://arxiv.org/abs/2204.02311v5","paper_title":"PaLM: Scaling Language Modeling with Pathways","code":"https://github.com/lucidrains/CoCa-pytorch","n_code_links":7,"syntology":{"n_ran":30,"n_unverified":7,"n_samples":37,"n_pointer_only_licence":0}},{"rank_in_archive_order":12,"model":"PaLM 540B (0-shot)","metrics":{"Accuracy":"89.1"},"uses_additional_data":false,"paper_date":"2022-04-05","paper":"/paper/palm-scaling-language-modeling-with-pathways-1","paper_url":"https://arxiv.org/abs/2204.02311v5","paper_title":"PaLM: Scaling Language Modeling with Pathways","code":"https://github.com/lucidrains/CoCa-pytorch","n_code_links":7,"syntology":{"n_ran":30,"n_unverified":7,"n_samples":37,"n_pointer_only_licence":0}},{"rank_in_archive_order":13,"model":"PaLM 2-M (1-shot)","metrics":{"Accuracy":"88.1"},"uses_additional_data":false,"paper_date":"2023-05-17","paper":"/paper/palm-2-technical-report-1","paper_url":"https://arxiv.org/abs/2305.10403v3","paper_title":"PaLM 2 Technical Report","code":"https://github.com/eternityyw/tram-benchmark","n_code_links":1,"syntology":null},{"rank_in_archive_order":14,"model":"PaLM 2-L (1-shot)","metrics":{"Accuracy":"86.9"},"uses_additional_data":false,"paper_date":"2023-05-17","paper":"/paper/palm-2-technical-report-1","paper_url":"https://arxiv.org/abs/2305.10403v3","paper_title":"PaLM 2 Technical Report","code":"https://github.com/eternityyw/tram-benchmark","n_code_links":1,"syntology":null},{"rank_in_archive_order":15,"model":"FLAN 137B (prompt-tuned)","metrics":{"Accuracy":"86.5"},"uses_additional_data":false,"paper_date":"2021-09-03","paper":"/paper/finetuned-language-models-are-zero-shot","paper_url":"https://arxiv.org/abs/2109.01652v5","paper_title":"Finetuned Language Models Are Zero-Shot Learners","code":"https://github.com/hiyouga/llama-efficient-tuning","n_code_links":8,"syntology":{"n_ran":0,"n_unverified":1,"n_samples":1,"n_pointer_only_licence":0}},{"rank_in_archive_order":16,"model":"PaLM 540B (1-shot)","metrics":{"Accuracy":"86.3"},"uses_additional_data":false,"paper_date":"2022-04-05","paper":"/paper/palm-scaling-language-modeling-with-pathways-1","paper_url":"https://arxiv.org/abs/2204.02311v5","paper_title":"PaLM: Scaling Language Modeling with Pathways","code":"https://github.com/lucidrains/CoCa-pytorch","n_code_links":7,"syntology":{"n_ran":30,"n_unverified":7,"n_samples":37,"n_pointer_only_licence":0}},{"rank_in_archive_order":17,"model":"TTTTT 3B (fine-tuned)","metrics":{"Accuracy":"84.6"},"uses_additional_data":false,"paper_date":"2020-03-18","paper":"/paper/tttttackling-winogrande-schemas","paper_url":"https://arxiv.org/abs/2003.08380v1","paper_title":"TTTTTackling WinoGrande Schemas","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":18,"model":"PaLM 2-S (1-shot)","metrics":{"Accuracy":"84.6"},"uses_additional_data":false,"paper_date":"2023-05-17","paper":"/paper/palm-2-technical-report-1","paper_url":"https://arxiv.org/abs/2305.10403v3","paper_title":"PaLM 2 Technical Report","code":"https://github.com/eternityyw/tram-benchmark","n_code_links":1,"syntology":null},{"rank_in_archive_order":19,"model":"RoBERTa-DPR 355M","metrics":{"Accuracy":"83.1"},"uses_additional_data":false,"paper_date":"2019-07-24","paper":"/paper/winogrande-an-adversarial-winograd-schema","paper_url":"https://arxiv.org/abs/1907.10641v2","paper_title":"WinoGrande: An Adversarial Winograd Schema Challenge at Scale","code":"https://github.com/vered1986/self_talk","n_code_links":10,"syntology":null},{"rank_in_archive_order":20,"model":"FLAN 137B (zero-shot)","metrics":{"Accuracy":"80.8"},"uses_additional_data":false,"paper_date":"2021-09-03","paper":"/paper/finetuned-language-models-are-zero-shot","paper_url":"https://arxiv.org/abs/2109.01652v5","paper_title":"Finetuned Language Models Are Zero-Shot Learners","code":"https://github.com/hiyouga/llama-efficient-tuning","n_code_links":8,"syntology":{"n_ran":0,"n_unverified":1,"n_samples":1,"n_pointer_only_licence":0}},{"rank_in_archive_order":21,"model":"GPT-3 175B (few-shot)","metrics":{"Accuracy":"80.1"},"uses_additional_data":false,"paper_date":"2020-05-28","paper":"/paper/language-models-are-few-shot-learners","paper_url":"https://arxiv.org/abs/2005.14165v4","paper_title":"Language Models are Few-Shot Learners","code":"https://github.com/ggml-org/llama.cpp","n_code_links":67,"syntology":{"n_ran":15,"n_unverified":50,"n_samples":65,"n_pointer_only_licence":4}},{"rank_in_archive_order":22,"model":"RoBERTa-large + G-DAug-Inf","metrics":{"Accuracy":"80"},"uses_additional_data":false,"paper_date":"2020-04-24","paper":"/paper/g-daug-generative-data-augmentation-for","paper_url":"https://arxiv.org/abs/2004.11546v3","paper_title":"Generative Data Augmentation for Commonsense Reasoning","code":"https://github.com/yangyiben/G-DAUG-c-Generative-Data-Augmentation-for-Commonsense-Reasoning","n_code_links":1,"syntology":{"n_ran":2,"n_unverified":2,"n_samples":4,"n_pointer_only_licence":4}},{"rank_in_archive_order":23,"model":"UL2 20B (0-shot)","metrics":{"Accuracy":"79.9"},"uses_additional_data":false,"paper_date":"2022-05-10","paper":"/paper/unifying-language-learning-paradigms","paper_url":"https://arxiv.org/abs/2205.05131v3","paper_title":"UL2: Unifying Language Learning Paradigms","code":"https://github.com/google-research/google-research","n_code_links":2,"syntology":{"n_ran":0,"n_unverified":16,"n_samples":16,"n_pointer_only_licence":0}},{"rank_in_archive_order":24,"model":"ALBERT-xxlarge 235M","metrics":{"Accuracy":"78.8"},"uses_additional_data":false,"paper_date":"2021-04-16","paper":"/paper/back-to-square-one-bias-detection-training","paper_url":"https://arxiv.org/abs/2104.08161v2","paper_title":"Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":25,"model":"Neo-6B (QA + WS)","metrics":{"Accuracy":"77.9"},"uses_additional_data":false,"paper_date":"2022-10-05","paper":"/paper/ask-me-anything-a-simple-strategy-for","paper_url":"https://arxiv.org/abs/2210.02441v3","paper_title":"Ask Me Anything: A simple strategy for prompting language models","code":"https://github.com/hazyresearch/ama_prompting","n_code_links":3,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":26,"model":"HNN","metrics":{"Accuracy":"75.1"},"uses_additional_data":false,"paper_date":"2019-07-27","paper":"/paper/a-hybrid-neural-network-model-for-commonsense","paper_url":"https://arxiv.org/abs/1907.11983v1","paper_title":"A Hybrid Neural Network Model for Commonsense Reasoning","code":"https://github.com/namisan/mt-dnn","n_code_links":3,"syntology":{"n_ran":0,"n_unverified":1,"n_samples":1,"n_pointer_only_licence":0}},{"rank_in_archive_order":27,"model":"Neo-6B (QA)","metrics":{"Accuracy":"74.7"},"uses_additional_data":false,"paper_date":"2022-10-05","paper":"/paper/ask-me-anything-a-simple-strategy-for","paper_url":"https://arxiv.org/abs/2210.02441v3","paper_title":"Ask Me Anything: A simple strategy for prompting language models","code":"https://github.com/hazyresearch/ama_prompting","n_code_links":3,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":28,"model":"RoBERTa-large 354M","metrics":{"Accuracy":"73.9"},"uses_additional_data":false,"paper_date":"2021-04-16","paper":"/paper/back-to-square-one-bias-detection-training","paper_url":"https://arxiv.org/abs/2104.08161v2","paper_title":"Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":29,"model":"GPT-2-XL 1.5B","metrics":{"Accuracy":"73.3"},"uses_additional_data":false,"paper_date":"2023-04-27","paper":"/paper/lamini-lm-a-diverse-herd-of-distilled-models","paper_url":"https://arxiv.org/abs/2304.14402v3","paper_title":"LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions","code":"https://github.com/mbzuai-nlp/lamini-lm","n_code_links":1,"syntology":null},{"rank_in_archive_order":30,"model":"BERTwiki 340M (fine-tuned on WSCR)","metrics":{"Accuracy":"72.5"},"uses_additional_data":false,"paper_date":"2019-05-15","paper":"/paper/a-surprisingly-robust-trick-for-winograd","paper_url":"https://arxiv.org/abs/1905.06290v2","paper_title":"A Surprisingly Robust Trick for Winograd Schema Challenge","code":"https://github.com/vid-koci/bert-commonsense","n_code_links":2,"syntology":null},{"rank_in_archive_order":31,"model":"BERT-SocialIQA 340M","metrics":{"Accuracy":"72.5"},"uses_additional_data":false,"paper_date":"2019-04-22","paper":"/paper/socialiqa-commonsense-reasoning-about-social","paper_url":"https://arxiv.org/abs/1904.09728v3","paper_title":"SocialIQA: Commonsense Reasoning about Social Interactions","code":"https://github.com/clear-nus/llm-human-model","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":10,"n_samples":10,"n_pointer_only_licence":0}},{"rank_in_archive_order":32,"model":"BERT-large 340M (fine-tuned on WSCR)","metrics":{"Accuracy":"71.4"},"uses_additional_data":false,"paper_date":"2019-05-15","paper":"/paper/a-surprisingly-robust-trick-for-winograd","paper_url":"https://arxiv.org/abs/1905.06290v2","paper_title":"A Surprisingly Robust Trick for Winograd Schema Challenge","code":"https://github.com/vid-koci/bert-commonsense","n_code_links":2,"syntology":null},{"rank_in_archive_order":33,"model":"GPT-2-XL 1.5B","metrics":{"Accuracy":"70.7"},"uses_additional_data":false,"paper_date":"2019-02-14","paper":"/paper/language-models-are-unsupervised-multitask","paper_url":"https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf","paper_title":"Language Models are Unsupervised Multitask Learners","code":"https://github.com/huggingface/transformers","n_code_links":21,"syntology":null},{"rank_in_archive_order":34,"model":"BERTwiki 340M (fine-tuned on half of WSCR)","metrics":{"Accuracy":"70.3"},"uses_additional_data":false,"paper_date":"2019-05-15","paper":"/paper/a-surprisingly-robust-trick-for-winograd","paper_url":"https://arxiv.org/abs/1905.06290v2","paper_title":"A Surprisingly Robust Trick for Winograd Schema Challenge","code":"https://github.com/vid-koci/bert-commonsense","n_code_links":2,"syntology":null},{"rank_in_archive_order":35,"model":"LaMini-GPT 1.5B","metrics":{"Accuracy":"69.6"},"uses_additional_data":false,"paper_date":"2023-04-27","paper":"/paper/lamini-lm-a-diverse-herd-of-distilled-models","paper_url":"https://arxiv.org/abs/2304.14402v3","paper_title":"LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions","code":"https://github.com/mbzuai-nlp/lamini-lm","n_code_links":1,"syntology":null},{"rank_in_archive_order":36,"model":"GPT-2 Medium 774M (partial scoring)","metrics":{"Accuracy":"69.2"},"uses_additional_data":false,"paper_date":"2018-11-05","paper":"/paper/on-the-evaluation-of-common-sense-reasoning","paper_url":"https://arxiv.org/abs/1811.01778v2","paper_title":"How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG","code":"https://github.com/ptrichel/How-Reasonable-are-Common-Sense-Reasoning-Tasks","n_code_links":1,"syntology":null},{"rank_in_archive_order":37,"model":"N-Grammer 343M","metrics":{"Accuracy":"68.3"},"uses_additional_data":false,"paper_date":"2022-07-13","paper":"/paper/n-grammer-augmenting-transformers-with-latent-1","paper_url":"https://arxiv.org/abs/2207.06366v1","paper_title":"N-Grammer: Augmenting Transformers with latent n-grams","code":"https://github.com/tensorflow/lingvo","n_code_links":2,"syntology":{"n_ran":0,"n_unverified":6,"n_samples":6,"n_pointer_only_licence":0}},{"rank_in_archive_order":38,"model":"AlexaTM 20B","metrics":{"Accuracy":"68.3"},"uses_additional_data":false,"paper_date":"2022-08-02","paper":"/paper/alexatm-20b-few-shot-learning-using-a-large","paper_url":"https://arxiv.org/abs/2208.01448v2","paper_title":"AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model","code":"https://github.com/amazon-science/alexa-teacher-models","n_code_links":1,"syntology":{"n_ran":1,"n_unverified":0,"n_samples":1,"n_pointer_only_licence":0}},{"rank_in_archive_order":39,"model":"BERT-large 340M","metrics":{"Accuracy":"67"},"uses_additional_data":false,"paper_date":"2019-04-22","paper":"/paper/socialiqa-commonsense-reasoning-about-social","paper_url":"https://arxiv.org/abs/1904.09728v3","paper_title":"SocialIQA: Commonsense Reasoning about Social Interactions","code":"https://github.com/clear-nus/llm-human-model","n_code_links":1,"syntology":{"n_ran":0,"n_unverified":10,"n_samples":10,"n_pointer_only_licence":0}},{"rank_in_archive_order":40,"model":"T5-Large 738M","metrics":{"Accuracy":"66.7"},"uses_additional_data":false,"paper_date":"2023-04-27","paper":"/paper/lamini-lm-a-diverse-herd-of-distilled-models","paper_url":"https://arxiv.org/abs/2304.14402v3","paper_title":"LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions","code":"https://github.com/mbzuai-nlp/lamini-lm","n_code_links":1,"syntology":null},{"rank_in_archive_order":41,"model":"T0-3B (CoT fine-tuned)","metrics":{"Accuracy":"66"},"uses_additional_data":false,"paper_date":"2023-05-23","paper":"/paper/the-cot-collection-improving-zero-shot-and","paper_url":"https://arxiv.org/abs/2305.14045v2","paper_title":"The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning","code":"https://github.com/kaistai/cot-collection","n_code_links":2,"syntology":null},{"rank_in_archive_order":42,"model":"KiC-770M","metrics":{"Accuracy":"65.40"},"uses_additional_data":false,"paper_date":"2022-10-28","paper":"/paper/knowledge-in-context-towards-knowledgeable","paper_url":"https://arxiv.org/abs/2210.16433v3","paper_title":"Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":43,"model":"GPT-2 Medium 774M (full scoring)","metrics":{"Accuracy":"64.5"},"uses_additional_data":false,"paper_date":"2018-11-05","paper":"/paper/on-the-evaluation-of-common-sense-reasoning","paper_url":"https://arxiv.org/abs/1811.01778v2","paper_title":"How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG","code":"https://github.com/ptrichel/How-Reasonable-are-Common-Sense-Reasoning-Tasks","n_code_links":1,"syntology":null},{"rank_in_archive_order":44,"model":"LaMini-F-T5 783M","metrics":{"Accuracy":"64.1"},"uses_additional_data":false,"paper_date":"2023-04-27","paper":"/paper/lamini-lm-a-diverse-herd-of-distilled-models","paper_url":"https://arxiv.org/abs/2304.14402v3","paper_title":"LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions","code":"https://github.com/mbzuai-nlp/lamini-lm","n_code_links":1,"syntology":null},{"rank_in_archive_order":45,"model":"Ensemble of 14 LMs","metrics":{"Accuracy":"63.7"},"uses_additional_data":false,"paper_date":"2018-06-07","paper":"/paper/a-simple-method-for-commonsense-reasoning","paper_url":"https://arxiv.org/abs/1806.02847v2","paper_title":"A Simple Method for Commonsense Reasoning","code":"https://github.com/tensorflow/models/tree/master/research/lm_commonsense","n_code_links":2,"syntology":{"n_ran":0,"n_unverified":7,"n_samples":7,"n_pointer_only_licence":0}},{"rank_in_archive_order":46,"model":"H3 125M (3-shot, rank classification)","metrics":{"Accuracy":"63.5"},"uses_additional_data":false,"paper_date":"2022-12-28","paper":"/paper/hungry-hungry-hippos-towards-language","paper_url":"https://arxiv.org/abs/2212.14052v3","paper_title":"Hungry Hungry Hippos: Towards Language Modeling with State Space Models","code":"https://github.com/hazyresearch/safari","n_code_links":3,"syntology":{"n_ran":7,"n_unverified":8,"n_samples":15,"n_pointer_only_licence":0}},{"rank_in_archive_order":47,"model":"DSSM","metrics":{"Accuracy":"63.0"},"uses_additional_data":false,"paper_date":"2019-04-03","paper":"/paper/unsupervised-deep-structured-semantic-models","paper_url":"http://arxiv.org/abs/1904.01938v1","paper_title":"Unsupervised Deep Structured Semantic Models for Commonsense Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":48,"model":"RoBERTa-base 125M","metrics":{"Accuracy":"63"},"uses_additional_data":false,"paper_date":"2021-04-16","paper":"/paper/back-to-square-one-bias-detection-training","paper_url":"https://arxiv.org/abs/2104.08161v2","paper_title":"Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":49,"model":"Word-level CNN+LSTM (partial scoring)","metrics":{"Accuracy":"62.6"},"uses_additional_data":false,"paper_date":"2018-06-07","paper":"/paper/a-simple-method-for-commonsense-reasoning","paper_url":"https://arxiv.org/abs/1806.02847v2","paper_title":"A Simple Method for Commonsense Reasoning","code":"https://github.com/tensorflow/models/tree/master/research/lm_commonsense","n_code_links":2,"syntology":{"n_ran":0,"n_unverified":7,"n_samples":7,"n_pointer_only_licence":0}},{"rank_in_archive_order":50,"model":"UDSSM-II (ensemble)","metrics":{"Accuracy":"62.4"},"uses_additional_data":false,"paper_date":"2019-04-03","paper":"/paper/unsupervised-deep-structured-semantic-models","paper_url":"http://arxiv.org/abs/1904.01938v1","paper_title":"Unsupervised Deep Structured Semantic Models for Commonsense Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":51,"model":"BERT-base 110M (fine-tuned on WSCR)","metrics":{"Accuracy":"62.3"},"uses_additional_data":false,"paper_date":"2019-05-15","paper":"/paper/a-surprisingly-robust-trick-for-winograd","paper_url":"https://arxiv.org/abs/1905.06290v2","paper_title":"A Surprisingly Robust Trick for Winograd Schema Challenge","code":"https://github.com/vid-koci/bert-commonsense","n_code_links":2,"syntology":null},{"rank_in_archive_order":52,"model":"RoE-3B","metrics":{"Accuracy":"62.21"},"uses_additional_data":false,"paper_date":"2023-02-07","paper":"/paper/exploring-the-benefits-of-training-expert","paper_url":"https://arxiv.org/abs/2302.03202v2","paper_title":"Exploring the Benefits of Training Expert Language Models over Instruction Tuning","code":"https://github.com/joeljang/rlphf","n_code_links":2,"syntology":{"n_ran":3,"n_unverified":0,"n_samples":3,"n_pointer_only_licence":3}},{"rank_in_archive_order":53,"model":"BERT-large 340M","metrics":{"Accuracy":"62.0"},"uses_additional_data":false,"paper_date":"2018-10-11","paper":"/paper/bert-pre-training-of-deep-bidirectional","paper_url":"https://arxiv.org/abs/1810.04805v2","paper_title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","code":"https://github.com/huggingface/transformers","n_code_links":534,"syntology":{"n_ran":204,"n_unverified":455,"n_samples":659,"n_pointer_only_licence":149}},{"rank_in_archive_order":54,"model":"GPT-2 Small 117M (partial scoring)","metrics":{"Accuracy":"61.5"},"uses_additional_data":false,"paper_date":"2018-11-05","paper":"/paper/on-the-evaluation-of-common-sense-reasoning","paper_url":"https://arxiv.org/abs/1811.01778v2","paper_title":"How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG","code":"https://github.com/ptrichel/How-Reasonable-are-Common-Sense-Reasoning-Tasks","n_code_links":1,"syntology":null},{"rank_in_archive_order":55,"model":"H3 125M (0-shot, rank classification)","metrics":{"Accuracy":"61.5"},"uses_additional_data":false,"paper_date":"2022-12-28","paper":"/paper/hungry-hungry-hippos-towards-language","paper_url":"https://arxiv.org/abs/2212.14052v3","paper_title":"Hungry Hungry Hippos: Towards Language Modeling with State Space Models","code":"https://github.com/hazyresearch/safari","n_code_links":3,"syntology":{"n_ran":7,"n_unverified":8,"n_samples":15,"n_pointer_only_licence":0}},{"rank_in_archive_order":56,"model":"BERT-large 340M","metrics":{"Accuracy":"61.4"},"uses_additional_data":false,"paper_date":"2021-04-16","paper":"/paper/back-to-square-one-bias-detection-training","paper_url":"https://arxiv.org/abs/2104.08161v2","paper_title":"Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":57,"model":"BERT-base 110M + MAS","metrics":{"Accuracy":"60.3"},"uses_additional_data":false,"paper_date":"2019-05-31","paper":"/paper/attention-is-not-all-you-need-for-commonsense","paper_url":"https://arxiv.org/abs/1905.13497v1","paper_title":"Attention Is (not) All You Need for Commonsense Reasoning","code":"https://github.com/SAP-samples/acl2020-commonsense","n_code_links":2,"syntology":null},{"rank_in_archive_order":58,"model":"longdoc S (OntoNotes + PreCo + LitBank)","metrics":{"Accuracy":"60.1"},"uses_additional_data":false,"paper_date":"2021-09-20","paper":"/paper/on-generalization-in-coreference-resolution","paper_url":"https://arxiv.org/abs/2109.09667v1","paper_title":"On Generalization in Coreference Resolution","code":"https://github.com/shtoshni/fast-coref","n_code_links":2,"syntology":null},{"rank_in_archive_order":59,"model":"longdoc S (ON + PreCo + LitBank + 30k pseudo-singletons)","metrics":{"Accuracy":"59.4"},"uses_additional_data":false,"paper_date":"2021-09-20","paper":"/paper/on-generalization-in-coreference-resolution","paper_url":"https://arxiv.org/abs/2109.09667v1","paper_title":"On Generalization in Coreference Resolution","code":"https://github.com/shtoshni/fast-coref","n_code_links":2,"syntology":null},{"rank_in_archive_order":60,"model":"UDSSM-II","metrics":{"Accuracy":"59.2"},"uses_additional_data":false,"paper_date":"2019-04-03","paper":"/paper/unsupervised-deep-structured-semantic-models","paper_url":"http://arxiv.org/abs/1904.01938v1","paper_title":"Unsupervised Deep Structured Semantic Models for Commonsense Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":61,"model":"LaMini-T5 738M","metrics":{"Accuracy":"59"},"uses_additional_data":false,"paper_date":"2023-04-27","paper":"/paper/lamini-lm-a-diverse-herd-of-distilled-models","paper_url":"https://arxiv.org/abs/2304.14402v3","paper_title":"LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions","code":"https://github.com/mbzuai-nlp/lamini-lm","n_code_links":1,"syntology":null},{"rank_in_archive_order":62,"model":"Flipped-3B","metrics":{"Accuracy":"58.37"},"uses_additional_data":false,"paper_date":"2022-10-06","paper":"/paper/guess-the-instruction-making-language-models","paper_url":"https://arxiv.org/abs/2210.02969v4","paper_title":"Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners","code":"https://github.com/seonghyeonye/flipped-learning","n_code_links":1,"syntology":null},{"rank_in_archive_order":63,"model":"KEE+NKAM winner of the WSC2016","metrics":{"Accuracy":"58.3"},"uses_additional_data":false,"paper_date":"2016-11-13","paper":"/paper/commonsense-knowledge-enhanced-embeddings-for","paper_url":"http://arxiv.org/abs/1611.04146v2","paper_title":"Commonsense Knowledge Enhanced Embeddings for Solving Pronoun Disambiguation Problems in Winograd Schema Challenge","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":64,"model":"Char-level CNN+LSTM (partial scoring)","metrics":{"Accuracy":"57.9"},"uses_additional_data":false,"paper_date":"2018-06-07","paper":"/paper/a-simple-method-for-commonsense-reasoning","paper_url":"https://arxiv.org/abs/1806.02847v2","paper_title":"A Simple Method for Commonsense Reasoning","code":"https://github.com/tensorflow/models/tree/master/research/lm_commonsense","n_code_links":2,"syntology":{"n_ran":0,"n_unverified":7,"n_samples":7,"n_pointer_only_licence":0}},{"rank_in_archive_order":65,"model":"UDSSM-I (ensemble)","metrics":{"Accuracy":"57.1"},"uses_additional_data":false,"paper_date":"2019-04-03","paper":"/paper/unsupervised-deep-structured-semantic-models","paper_url":"http://arxiv.org/abs/1904.01938v1","paper_title":"Unsupervised Deep Structured Semantic Models for Commonsense Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":66,"model":"Knowledge Hunter","metrics":{"Accuracy":"57.1"},"uses_additional_data":false,"paper_date":"2018-10-02","paper":"/paper/a-knowledge-hunting-framework-for-common","paper_url":"http://arxiv.org/abs/1810.01375v1","paper_title":"A Knowledge Hunting Framework for Common Sense Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":67,"model":"WKH","metrics":{"Accuracy":"57.1"},"uses_additional_data":false,"paper_date":"2019-07-24","paper":"/paper/winogrande-an-adversarial-winograd-schema","paper_url":"https://arxiv.org/abs/1907.10641v2","paper_title":"WinoGrande: An Adversarial Winograd Schema Challenge at Scale","code":"https://github.com/vered1986/self_talk","n_code_links":10,"syntology":null},{"rank_in_archive_order":68,"model":"BERT-base 110M","metrics":{"Accuracy":"56.5"},"uses_additional_data":false,"paper_date":"2021-04-16","paper":"/paper/back-to-square-one-bias-detection-training","paper_url":"https://arxiv.org/abs/2104.08161v2","paper_title":"Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":69,"model":"GPT-2 Small 117M (full scoring)","metrics":{"Accuracy":"55.7"},"uses_additional_data":false,"paper_date":"2018-11-05","paper":"/paper/on-the-evaluation-of-common-sense-reasoning","paper_url":"https://arxiv.org/abs/1811.01778v2","paper_title":"How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG","code":"https://github.com/ptrichel/How-Reasonable-are-Common-Sense-Reasoning-Tasks","n_code_links":1,"syntology":null},{"rank_in_archive_order":70,"model":"ALBERT-base 11M","metrics":{"Accuracy":"55.4"},"uses_additional_data":false,"paper_date":"2021-04-16","paper":"/paper/back-to-square-one-bias-detection-training","paper_url":"https://arxiv.org/abs/2104.08161v2","paper_title":"Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":71,"model":"Pythia 12B (0-shot)","metrics":{"Accuracy":"54.8"},"uses_additional_data":false,"paper_date":"2023-04-03","paper":"/paper/pythia-a-suite-for-analyzing-large-language","paper_url":"https://arxiv.org/abs/2304.01373v2","paper_title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","code":"https://github.com/Lightning-AI/lit-gpt","n_code_links":4,"syntology":null},{"rank_in_archive_order":72,"model":"UDSSM-I","metrics":{"Accuracy":"54.5"},"uses_additional_data":false,"paper_date":"2019-04-03","paper":"/paper/unsupervised-deep-structured-semantic-models","paper_url":"http://arxiv.org/abs/1904.01938v1","paper_title":"Unsupervised Deep Structured Semantic Models for Commonsense Reasoning","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":73,"model":"Subword-level Transformer LM","metrics":{"Accuracy":"54.1"},"uses_additional_data":false,"paper_date":"2017-06-12","paper":"/paper/attention-is-all-you-need","paper_url":"https://arxiv.org/abs/1706.03762v7","paper_title":"Attention Is All You Need","code":"https://github.com/huggingface/transformers","n_code_links":595,"syntology":{"n_ran":600,"n_unverified":346,"n_samples":946,"n_pointer_only_licence":451}},{"rank_in_archive_order":74,"model":"USSM + Supervised DeepNet + KB","metrics":{"Accuracy":"52.8"},"uses_additional_data":false,"paper_date":"2019-05-31","paper":"/paper/attention-is-not-all-you-need-for-commonsense","paper_url":"https://arxiv.org/abs/1905.13497v1","paper_title":"Attention Is (not) All You Need for Commonsense Reasoning","code":"https://github.com/SAP-samples/acl2020-commonsense","n_code_links":2,"syntology":null},{"rank_in_archive_order":75,"model":"KEE+NKAM on WinoGrande","metrics":{"Accuracy":"52.8"},"uses_additional_data":false,"paper_date":"2019-07-24","paper":"/paper/winogrande-an-adversarial-winograd-schema","paper_url":"https://arxiv.org/abs/1907.10641v2","paper_title":"WinoGrande: An Adversarial Winograd Schema Challenge at Scale","code":"https://github.com/vered1986/self_talk","n_code_links":10,"syntology":null},{"rank_in_archive_order":76,"model":"USSM + KB","metrics":{"Accuracy":"52"},"uses_additional_data":false,"paper_date":"2019-05-31","paper":"/paper/attention-is-not-all-you-need-for-commonsense","paper_url":"https://arxiv.org/abs/1905.13497v1","paper_title":"Attention Is (not) All You Need for Commonsense Reasoning","code":"https://github.com/SAP-samples/acl2020-commonsense","n_code_links":2,"syntology":null},{"rank_in_archive_order":77,"model":"Random chance baseline","metrics":{"Accuracy":"50"},"uses_additional_data":false,"paper_date":"2021-04-16","paper":"/paper/back-to-square-one-bias-detection-training","paper_url":"https://arxiv.org/abs/2104.08161v2","paper_title":"Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":78,"model":"Hybrid H3 125M (3-shot, logit scoring)","metrics":{"Accuracy":"43.3"},"uses_additional_data":false,"paper_date":"2022-12-28","paper":"/paper/hungry-hungry-hippos-towards-language","paper_url":"https://arxiv.org/abs/2212.14052v3","paper_title":"Hungry Hungry Hippos: Towards Language Modeling with State Space Models","code":"https://github.com/hazyresearch/safari","n_code_links":3,"syntology":{"n_ran":7,"n_unverified":8,"n_samples":15,"n_pointer_only_licence":0}},{"rank_in_archive_order":79,"model":"Pythia 2.8B (0-shot)","metrics":{"Accuracy":"38.5"},"uses_additional_data":false,"paper_date":"2023-04-03","paper":"/paper/pythia-a-suite-for-analyzing-large-language","paper_url":"https://arxiv.org/abs/2304.01373v2","paper_title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","code":"https://github.com/Lightning-AI/lit-gpt","n_code_links":4,"syntology":null},{"rank_in_archive_order":80,"model":"Neo-6B (few-shot)","metrics":{"Accuracy":"36.5"},"uses_additional_data":false,"paper_date":"2022-10-05","paper":"/paper/ask-me-anything-a-simple-strategy-for","paper_url":"https://arxiv.org/abs/2210.02441v3","paper_title":"Ask Me Anything: A simple strategy for prompting language models","code":"https://github.com/hazyresearch/ama_prompting","n_code_links":3,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":0}},{"rank_in_archive_order":81,"model":"Pythia 6.9B (0-shot)","metrics":{"Accuracy":"36.5"},"uses_additional_data":false,"paper_date":"2023-04-03","paper":"/paper/pythia-a-suite-for-analyzing-large-language","paper_url":"https://arxiv.org/abs/2304.01373v2","paper_title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","code":"https://github.com/Lightning-AI/lit-gpt","n_code_links":4,"syntology":null},{"rank_in_archive_order":82,"model":"Pythia 12B (5-shot)","metrics":{"Accuracy":"36.5"},"uses_additional_data":false,"paper_date":"2023-04-03","paper":"/paper/pythia-a-suite-for-analyzing-large-language","paper_url":"https://arxiv.org/abs/2304.01373v2","paper_title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","code":"https://github.com/Lightning-AI/lit-gpt","n_code_links":4,"syntology":null}],"since_archive":{"claim":"Results that newer papers report for their own method, placed here by Syntology. A model pointed at the cell in the paper's own table; the number was read from that cell and checked against this leaderboard's metric, dataset, split and scale; an independent check that saw this leaderboard's other rows and every other leaderboard on the same dataset accepted it. Not reviewed by the paper's authors or by the archive's editors, and not ranked against the archive rows.","extraction_file_present":true,"measurement":{"test_papers":883,"papers_with_output":881,"judged_true":108,"judged":110,"wilson95_lower":0.9361,"measured_on":"2026-09-24","frozen_commit":"0e3de0df94"},"measurement_note":"blind adjudication of accepted entries on a held-out split of archive papers, rules frozen before the test","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","complete":false,"papers_newer_than_archive":9581,"papers_checked":6264,"papers_extracted_not_yet_verified":0,"boards_without_verdict":2,"papers_not_yet_extracted":3316},"order":"newest first by month (arXiv date, else the arXiv-id month), then arXiv id descending","columns":[],"entries":[]},"syntology":{"read_at":"2026-09-24T18:15:14+00:00","claim":"Per row: N of M harvested code samples from that row's paper executed on a synthesized fixture; the other M-N are unverified. Not a reproduction of the row's number; not a correctness claim. n_pointer_only_licence counts samples the site points at rather than redistributes (a licence axis, independent of ran/unverified).","rows_with_graph_line":32,"rows_with_any_sample_ran":21,"distinct_papers_with_graph_line":19,"distinct_papers_with_any_sample_ran":13,"samples_over_distinct_papers":{"n_ran":883,"n_unverified":956,"n_samples":1839,"n_pointer_only_licence":621,"note":"each paper (arXiv id) counted once, however many rows it is behind; this is the page-level figure"},"samples_row_weighted":{"n_ran":996,"n_unverified":1034,"n_samples":2030,"n_pointer_only_licence":626,"note":"row-weighted: a paper behind several rows is counted once per row; inflated relative to samples_over_distinct_papers by design, kept for readers summing the per-row syntology blocks"}}}