{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bringing-light-into-the-dark-a-large-scale","title":"Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework","arxiv_id":"2006.13365","date":"2020-06-23","proceeding":null,"authors":["Mehdi Ali","Max Berrendorf","Charles Tapley Hoyt","Laurent Vermue","Mikhail Galkin","Sahand Sharifzadeh","Asja Fischer","Volker Tresp","Jens Lehmann"],"abstract":"The heterogeneity in recently published knowledge graph embedding models' implementations, training, and evaluation has made fair and thorough comparisons difficult. In order to assess the reproducibility of previously published results, we re-implemented and evaluated 21 interaction models in the PyKEEN software package. Here, we outline which results could be reproduced with their reported hyper-parameters, which could only be reproduced with alternate hyper-parameters, and which could not be reproduced at all as well as provide insight as to why this might be the case. We then performed a large-scale benchmarking on four datasets with several thousands of experiments and 24,804 GPU hours of computation time. We present insights gained as to best practices, best configurations for each model, and where improvements could be made over previously published best configurations. Our results highlight that the combination of model architecture, training approach, loss function, and the explicit modeling of inverse relations is crucial for a model's performances, and not only determined by the model architecture. We provide evidence that several architectures can obtain results competitive to the state-of-the-art when configured carefully. We have made all code, experimental configurations, results, and analyses that lead to our interpretations available at https://github.com/pykeen/pykeen and https://github.com/pykeen/benchmarking","url_abs":"https://arxiv.org/abs/2006.13365v5","url_pdf":"https://arxiv.org/pdf/2006.13365v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bringing-light-into-the-dark-a-large-scale","repo_url":"https://github.com/pykeen/benchmarking","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"bringing-light-into-the-dark-a-large-scale","repo_url":"https://github.com/SmartDataAnalytics/PyKEEN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"graph-embedding","task_name":"Graph Embedding"},{"task_slug":"knowledge-graph-embedding","task_name":"Knowledge Graph Embedding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.13365","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.13365"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pykeen/benchmarking","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SmartDataAnalytics/PyKEEN","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"267943e1ccc309ec","entry":"get_latex","repo":"pykeen/benchmarking","repo_kind":"official","path":"reproducibility/generate_summary_table.py","file_url":"https://github.com/pykeen/benchmarking/blob/HEAD/reproducibility/generate_summary_table.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"267943e1ccc309ec"}},{"code_sha256_prefix":"f2463ed829c4ee90","entry":"get_skyline_df","repo":"pykeen/benchmarking","repo_kind":"official","path":"src/pykeen_report/plot.py","file_url":"https://github.com/pykeen/benchmarking/blob/HEAD/src/pykeen_report/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2463ed829c4ee90"}},{"code_sha256_prefix":"efb9becff1a20e63","entry":"make_config_index","repo":"pykeen/benchmarking","repo_kind":"official","path":"src/pykeen_report/plot.py","file_url":"https://github.com/pykeen/benchmarking/blob/HEAD/src/pykeen_report/plot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"efb9becff1a20e63"}},{"code_sha256_prefix":"0912b172bcd759b8","entry":"make_index","repo":"pykeen/benchmarking","repo_kind":"official","path":"ablation/interpretation_tables.py","file_url":"https://github.com/pykeen/benchmarking/blob/HEAD/ablation/interpretation_tables.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0912b172bcd759b8"}},{"code_sha256_prefix":"723b75c137c4e378","entry":"read_ablation_collation","repo":"pykeen/benchmarking","repo_kind":"official","path":"src/pykeen_report/utils.py","file_url":"https://github.com/pykeen/benchmarking/blob/HEAD/src/pykeen_report/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"723b75c137c4e378"}},{"code_sha256_prefix":"71a3002f4eb5b01d","entry":"reorganize_summary_df","repo":"pykeen/benchmarking","repo_kind":"official","path":"reproducibility/generate_summary_table.py","file_url":"https://github.com/pykeen/benchmarking/blob/HEAD/reproducibility/generate_summary_table.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71a3002f4eb5b01d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}