{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reclor-a-reading-comprehension-dataset-1","title":"ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning","arxiv_id":"2002.04326","date":"2020-02-11","proceeding":"ICLR 2020 1","authors":["Weihao Yu","Zi-Hang Jiang","Yanfei Dong","Jiashi Feng"],"abstract":"Recent powerful pre-trained language models have achieved remarkable performance on most of the popular datasets for reading comprehension. It is time to introduce more challenging datasets to push the development of this field towards more comprehensive reasoning of text. In this paper, we introduce a new Reading Comprehension dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. As earlier studies suggest, human-annotated datasets usually contain biases, which are often exploited by models to achieve high accuracy without truly understanding the text. In order to comprehensively evaluate the logical reasoning ability of models on ReClor, we propose to identify biased data points and separate them into EASY set while the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.","url_abs":"https://arxiv.org/abs/2002.04326v3","url_pdf":"https://arxiv.org/pdf/2002.04326v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reclor-a-reading-comprehension-dataset-1","repo_url":"https://github.com/yuweihao/reclor","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"},{"task_slug":"logical-reasoning-question-ansering","task_name":"Logical Reasoning Question Answering"},{"task_slug":"logical-reasoning-reading-comprehension","task_name":"Logical Reasoning Reading Comprehension"},{"task_slug":"machine-reading-comprehension","task_name":"Machine Reading Comprehension"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"}],"methods":[],"datasets_introduced":[{"slug":"reclor","name":"ReClor","full_name":""}],"methods_introduced":[],"results":[{"leaderboard":"/sota/logical-reasoning-question-ansering-on-reclor","task":"Logical Reasoning Question Answering","dataset":"ReClor","model":"XLNet-large","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy":"56.0","Accuracy (easy)":"75.7","Accuracy (hard)":"40.5"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-question-ansering-on-reclor","task":"Logical Reasoning Question Answering","dataset":"ReClor","model":"RoBERTa-large","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy":"55.6","Accuracy (easy)":"75.5","Accuracy (hard)":"40.0"},"uses_additional_data":false},{"leaderboard":"/sota/logical-reasoning-question-ansering-on-reclor","task":"Logical Reasoning Question Answering","dataset":"ReClor","model":"BERT-large","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy":"49.8","Accuracy (easy)":"72.0","Accuracy (hard)":"32.3"},"uses_additional_data":false},{"leaderboard":"/sota/machine-reading-comprehension-on-reclor","task":"Machine Reading Comprehension","dataset":"ReClor","model":"XLNet-large","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy":"56.0","Accuracy (easy)":"75.7","Accuracy (hard)":"40.5"},"uses_additional_data":false},{"leaderboard":"/sota/machine-reading-comprehension-on-reclor","task":"Machine Reading Comprehension","dataset":"ReClor","model":"RoBERTa-large","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy":"55.6","Accuracy (easy)":"75.5","Accuracy (hard)":"40.0"},"uses_additional_data":false},{"leaderboard":"/sota/machine-reading-comprehension-on-reclor","task":"Machine Reading Comprehension","dataset":"ReClor","model":"BERT-large","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy":"49.8","Accuracy (easy)":"72.0","Accuracy (hard)":"32.3"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-reclor","task":"Question Answering","dataset":"ReClor","model":"XLNet-large","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy":"56.0","Accuracy (easy)":"75.7","Accuracy (hard)":"40.5"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-reclor","task":"Question Answering","dataset":"ReClor","model":"RoBERTa-large","rank_in_archive_order":2,"of":3,"metrics":{"Accuracy":"55.6","Accuracy (easy)":"75.5","Accuracy (hard)":"40.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-reclor","task":"Question Answering","dataset":"ReClor","model":"BERT-large","rank_in_archive_order":3,"of":3,"metrics":{"Accuracy":"49.8","Accuracy (easy)":"72.0","Accuracy (hard)":"32.3"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-reclor","task":"Reading Comprehension","dataset":"ReClor","model":"XLNet-large","rank_in_archive_order":30,"of":39,"metrics":{"Test":"56.0"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-reclor","task":"Reading Comprehension","dataset":"ReClor","model":"XLNet-base","rank_in_archive_order":32,"of":39,"metrics":{"Test":"50.4"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-reclor","task":"Reading Comprehension","dataset":"ReClor","model":"RoBERTa-base","rank_in_archive_order":35,"of":39,"metrics":{"Test":"48.5"},"uses_additional_data":false},{"leaderboard":"/sota/reading-comprehension-on-reclor","task":"Reading Comprehension","dataset":"ReClor","model":"BERT-base","rank_in_archive_order":36,"of":39,"metrics":{"Test":"47.3"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2002.04326","atlas_url":"https://app.syntology.ai/?focus=2002.04326","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2002.04326"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yuweihao/reclor","reach":null}],"summary":{"unverified":2},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"0a546b305d274996","entry":"select_field","repo":"yuweihao/reclor","repo_kind":"official","path":"run_multiple_choice.py","file_url":"https://github.com/yuweihao/reclor/blob/HEAD/run_multiple_choice.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a546b305d274996"}},{"code_sha256_prefix":"3c241ecfe3749a6d","entry":"simple_accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3c241ecfe3749a6d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}