{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/copen-probing-conceptual-knowledge-in-pre","title":"COPEN: Probing Conceptual Knowledge in Pre-trained Language Models","arxiv_id":"2211.04079","date":"2022-11-08","proceeding":null,"authors":["Hao Peng","Xiaozhi Wang","Shengding Hu","Hailong Jin","Lei Hou","Juanzi Li","Zhiyuan Liu","Qun Liu"],"abstract":"Conceptual knowledge is fundamental to human cognition and knowledge bases. However, existing knowledge probing works only focus on evaluating factual knowledge of pre-trained language models (PLMs) and ignore conceptual knowledge. Since conceptual knowledge often appears as implicit commonsense behind texts, designing probes for conceptual knowledge is hard. Inspired by knowledge representation schemata, we comprehensively evaluate conceptual knowledge of PLMs by designing three tasks to probe whether PLMs organize entities by conceptual similarities, learn conceptual properties, and conceptualize entities in contexts, respectively. For the tasks, we collect and annotate 24k data instances covering 393 concepts, which is COPEN, a COnceptual knowledge Probing bENchmark. Extensive experiments on different sizes and types of PLMs show that existing PLMs systematically lack conceptual knowledge and suffer from various spurious correlations. We believe this is a critical bottleneck for realizing human-like cognition in PLMs. COPEN and our codes are publicly released at https://github.com/THU-KEG/COPEN.","url_abs":"https://arxiv.org/abs/2211.04079v1","url_pdf":"https://arxiv.org/pdf/2211.04079v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"copen-probing-conceptual-knowledge-in-pre","repo_url":"https://github.com/thu-keg/copen","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-probing","task_name":"Knowledge Probing"}],"methods":[],"datasets_introduced":[{"slug":"copen","name":"COPEN","full_name":"COnceptual knowledge Probing bENchmark"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2211.04079","atlas_url":"https://app.syntology.ai/?focus=2211.04079","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2211.04079"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-keg/copen","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/THU-KEG/COPEN","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3e9b602ddddea5cd","entry":"compute_F1","repo":"THU-KEG/COPEN","repo_kind":"official","path":"code/finetuning/metrics.py","file_url":"https://github.com/THU-KEG/COPEN/blob/HEAD/code/finetuning/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9b602ddddea5cd"}},{"code_sha256_prefix":"0a98f376f88ee0c4","entry":"compute_accuracy","repo":"THU-KEG/COPEN","repo_kind":"official","path":"code/finetuning/metrics.py","file_url":"https://github.com/THU-KEG/COPEN/blob/HEAD/code/finetuning/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a98f376f88ee0c4"}},{"code_sha256_prefix":"aa0f399def18c806","entry":"softmax","repo":"THU-KEG/COPEN","repo_kind":"official","path":"code/finetuning/metrics.py","file_url":"https://github.com/THU-KEG/COPEN/blob/HEAD/code/finetuning/metrics.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa0f399def18c806"}},{"code_sha256_prefix":"491b240d8441f92b","entry":"get_model_for_mc","repo":"THU-KEG/COPEN","repo_kind":"official","path":"code/finetuning/model.py","file_url":"https://github.com/THU-KEG/COPEN/blob/HEAD/code/finetuning/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"491b240d8441f92b"}},{"code_sha256_prefix":"ec666b70f8033335","entry":"get_model_for_qa","repo":"THU-KEG/COPEN","repo_kind":"official","path":"code/finetuning/model.py","file_url":"https://github.com/THU-KEG/COPEN/blob/HEAD/code/finetuning/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec666b70f8033335"}},{"code_sha256_prefix":"27683166677284e2","entry":"get_model_for_sc","repo":"THU-KEG/COPEN","repo_kind":"official","path":"code/finetuning/model.py","file_url":"https://github.com/THU-KEG/COPEN/blob/HEAD/code/finetuning/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27683166677284e2"}},{"code_sha256_prefix":"2baa637bd7ab0470","entry":"prepare_data_for_model_size","repo":"THU-KEG/COPEN","repo_kind":"official","path":"code/visualize.py","file_url":"https://github.com/THU-KEG/COPEN/blob/HEAD/code/visualize.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2baa637bd7ab0470"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}