{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisiting-inferential-benchmarks-for","title":"Revisiting Inferential Benchmarks for Knowledge Graph Completion","arxiv_id":"2306.04814","date":"2023-06-07","proceeding":null,"authors":["Shuwen Liu","Bernardo Cuenca Grau","Ian Horrocks","Egor V. Kostylev"],"abstract":"Knowledge Graph (KG) completion is the problem of extending an incomplete KG with missing facts. A key feature of Machine Learning approaches for KG completion is their ability to learn inference patterns, so that the predicted facts are the results of applying these patterns to the KG. Standard completion benchmarks, however, are not well-suited for evaluating models' abilities to learn patterns, because the training and test sets of these benchmarks are a random split of a given KG and hence do not capture the causality of inference patterns. We propose a novel approach for designing KG completion benchmarks based on the following principles: there is a set of logical rules so that the missing facts are the results of the rules' application; the training set includes both premises matching rule antecedents and the corresponding conclusions; the test set consists of the results of applying the rules to the training set; the negative examples are designed to discourage the models from learning rules not entailed by the rule set. We use our methodology to generate several benchmarks and evaluate a wide range of existing KG completion systems. Our results provide novel insights on the ability of existing models to induce inference patterns from incomplete KGs.","url_abs":"https://arxiv.org/abs/2306.04814v1","url_pdf":"https://arxiv.org/pdf/2306.04814v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revisiting-inferential-benchmarks-for","repo_url":"https://github.com/shuwen-liu-ox/loginfer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"knowledge-graph-completion","task_name":"Knowledge Graph Completion"}],"methods":[{"method_slug":"test","method_name":"Test"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.04814","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.04814"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shuwen-liu-ox/loginfer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c22f3b1ca2468f9f","entry":"dictionary","repo":"shuwen-liu-ox/loginfer","repo_kind":"official","path":"generate_command.py","file_url":"https://github.com/shuwen-liu-ox/loginfer/blob/HEAD/generate_command.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c22f3b1ca2468f9f"}},{"code_sha256_prefix":"a0b9dcca830a021f","entry":"get_file_len","repo":"shuwen-liu-ox/loginfer","repo_kind":"official","path":"split.py","file_url":"https://github.com/shuwen-liu-ox/loginfer/blob/HEAD/split.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a0b9dcca830a021f"}},{"code_sha256_prefix":"e334987e22ac0fe6","entry":"read_assignments","repo":"shuwen-liu-ox/loginfer","repo_kind":"official","path":"negative_qg.py","file_url":"https://github.com/shuwen-liu-ox/loginfer/blob/HEAD/negative_qg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e334987e22ac0fe6"}},{"code_sha256_prefix":"bb58f9581037b4f5","entry":"read_assignments2","repo":"shuwen-liu-ox/loginfer","repo_kind":"official","path":"negative_qg.py","file_url":"https://github.com/shuwen-liu-ox/loginfer/blob/HEAD/negative_qg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb58f9581037b4f5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}