{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/large-scale-cloze-test-dataset-created-by","title":"Large-scale Cloze Test Dataset Created by Teachers","arxiv_id":"1711.03225","date":"2017-11-09","proceeding":"EMNLP 2018 10","authors":["Qizhe Xie","Guokun Lai","Zihang Dai","Eduard Hovy"],"abstract":"Cloze tests are widely adopted in language exams to evaluate students'\nlanguage proficiency. In this paper, we propose the first large-scale\nhuman-created cloze test dataset CLOTH, containing questions used in\nmiddle-school and high-school language exams. With missing blanks carefully\ncreated by teachers and candidate choices purposely designed to be nuanced,\nCLOTH requires a deeper language understanding and a wider attention span than\npreviously automatically-generated cloze datasets. We test the performance of\ndedicatedly designed baseline models including a language model trained on the\nOne Billion Word Corpus and show humans outperform them by a significant\nmargin. We investigate the source of the performance gap, trace model\ndeficiencies to some distinct properties of CLOTH, and identify the limited\nability of comprehending the long-term context to be the key bottleneck.","url_abs":"http://arxiv.org/abs/1711.03225v3","url_pdf":"http://arxiv.org/pdf/1711.03225v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"cloze-test","task_name":"Cloze Test"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[{"slug":"cloth","name":"CLOTH","full_name":"CLOze test by TeacHers"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1711.03225","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}