{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/winowhy-a-deep-diagnosis-of-essential","title":"WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge","arxiv_id":"2005.05763","date":"2020-05-12","proceeding":"ACL 2020 6","authors":["Hongming Zhang","Xinran Zhao","Yangqiu Song"],"abstract":"In this paper, we present the first comprehensive categorization of essential commonsense knowledge for answering the Winograd Schema Challenge (WSC). For each of the questions, we invite annotators to first provide reasons for making correct decisions and then categorize them into six major knowledge categories. By doing so, we better understand the limitation of existing methods (i.e., what kind of knowledge cannot be effectively represented or inferred with existing methods) and shed some light on the commonsense knowledge that we need to acquire in the future for better commonsense reasoning. Moreover, to investigate whether current WSC models can understand the commonsense or they simply solve the WSC questions based on the statistical bias of the dataset, we leverage the collected reasons to develop a new task called WinoWhy, which requires models to distinguish plausible reasons from very similar but wrong reasons for all WSC questions. Experimental results prove that even though pre-trained language representation models have achieved promising progress on the original WSC dataset, they are still struggling at WinoWhy. Further experiments show that even though supervised models can achieve better performance, the performance of these models can be sensitive to the dataset distribution. WinoWhy and all codes are available at: https://github.com/HKUST-KnowComp/WinoWhy.","url_abs":"https://arxiv.org/abs/2005.05763v1","url_pdf":"https://arxiv.org/pdf/2005.05763v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"winowhy-a-deep-diagnosis-of-essential","repo_url":"https://github.com/HKUST-KnowComp/WinoWhy","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"winowhy","task_name":"Winowhy"}],"methods":[],"datasets_introduced":[{"slug":"winowhy","name":"WinoWhy","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2005.05763","atlas_url":"https://app.syntology.ai/?focus=2005.05763","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2005.05763"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HKUST-KnowComp/WinoWhy","reach":null}],"summary":{"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"48445ba2525e6274","entry":"pick_training_and_testing_folds","repo":"HKUST-KnowComp/WinoWhy","repo_kind":"official","path":"supervised_winowhy.py","file_url":"https://github.com/HKUST-KnowComp/WinoWhy/blob/HEAD/supervised_winowhy.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48445ba2525e6274"}},{"code_sha256_prefix":"7479c61103f75348","entry":"output_five_folds","repo":"HKUST-KnowComp/WinoWhy","repo_kind":"official","path":"supervised_winowhy.py","file_url":"https://github.com/HKUST-KnowComp/WinoWhy/blob/HEAD/supervised_winowhy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7479c61103f75348"}},{"code_sha256_prefix":"e210d3e72ed13610","entry":"test","repo":"HKUST-KnowComp/WinoWhy","repo_kind":"official","path":"supervised_winowhy.py","file_url":"https://github.com/HKUST-KnowComp/WinoWhy/blob/HEAD/supervised_winowhy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e210d3e72ed13610"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}