{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/2309-05936","title":"Do PLMs Know and Understand Ontological Knowledge?","arxiv_id":"2309.05936","date":"2023-09-12","proceeding":null,"authors":["Weiqi Wu","Chengyue Jiang","Yong Jiang","Pengjun Xie","Kewei Tu"],"abstract":"Ontological knowledge, which comprises classes and properties and their relationships, is integral to world knowledge. It is significant to explore whether Pretrained Language Models (PLMs) know and understand such knowledge. However, existing PLM-probing studies focus mainly on factual knowledge, lacking a systematic probing of ontological knowledge. In this paper, we focus on probing whether PLMs store ontological knowledge and have a semantic understanding of the knowledge rather than rote memorization of the surface form. To probe whether PLMs know ontological knowledge, we investigate how well PLMs memorize: (1) types of entities; (2) hierarchical relationships among classes and properties, e.g., Person is a subclass of Animal and Member of Sports Team is a subproperty of Member of ; (3) domain and range constraints of properties, e.g., the subject of Member of Sports Team should be a Person and the object should be a Sports Team. To further probe whether PLMs truly understand ontological knowledge beyond memorization, we comprehensively study whether they can reliably perform logical reasoning with given knowledge according to ontological entailment rules. Our probing results show that PLMs can memorize certain ontological knowledge and utilize implicit knowledge in reasoning. However, both the memorizing and reasoning performances are less than perfect, indicating incomplete knowledge and understanding.","url_abs":"https://arxiv.org/abs/2309.05936v1","url_pdf":"https://arxiv.org/pdf/2309.05936v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"2309-05936","repo_url":"https://github.com/vickywu1022/ontoprobe-plms","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"logical-reasoning","task_name":"Logical Reasoning"},{"task_slug":"memorization","task_name":"Memorization"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2309.05936","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.05936"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vickywu1022/ontoprobe-plms","reach":{"status":"ok"}}],"summary":{"ran":8,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"fb31cb7a3029c5a7","entry":"MRR","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"metric.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb31cb7a3029c5a7"}},{"code_sha256_prefix":"a298647943f0dd08","entry":"MRR_avg","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"metric.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a298647943f0dd08"}},{"code_sha256_prefix":"8ac78d7341d5459e","entry":"domain_manual_textb","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"test-gpt/probing_gpt.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/test-gpt/probing_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ac78d7341d5459e"}},{"code_sha256_prefix":"67d3518e92b4271a","entry":"get_batch_curr_num_token","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"probing.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/probing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67d3518e92b4271a"}},{"code_sha256_prefix":"2dc0e6274c60ffb3","entry":"group_logits","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"probing.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/probing.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2dc0e6274c60ffb3"}},{"code_sha256_prefix":"e23b43db014c438f","entry":"range_manual_textb","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"test-gpt/probing_gpt.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/test-gpt/probing_gpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e23b43db014c438f"}},{"code_sha256_prefix":"211adb57bd1a1067","entry":"split_by_Acc","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"get_memorized.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/get_memorized.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"211adb57bd1a1067"}},{"code_sha256_prefix":"de6cfddda3efcae3","entry":"split_by_MRR","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"get_memorized.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/get_memorized.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"de6cfddda3efcae3"}},{"code_sha256_prefix":"5249b5134477a0e2","entry":"evaluate_batch","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"metric.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5249b5134477a0e2"}},{"code_sha256_prefix":"781f228a2f7d1135","entry":"load_candidates","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"data.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"781f228a2f7d1135"}},{"code_sha256_prefix":"a2f9f1bb06efa8bb","entry":"load_soft_emb_with_conj","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"utils.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2f9f1bb06efa8bb"}},{"code_sha256_prefix":"0225cb3676246cd4","entry":"load_soft_emb_without_conj","repo":"vickywu1022/ontoprobe-plms","repo_kind":"official","path":"utils.py","file_url":"https://github.com/vickywu1022/ontoprobe-plms/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0225cb3676246cd4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}