{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/language-models-are-open-knowledge-graphs-1","title":"Language Models are Open Knowledge Graphs","arxiv_id":"2010.11967","date":"2020-10-22","proceeding":null,"authors":["Chenguang Wang","Xiao Liu","Dawn Song"],"abstract":"This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.","url_abs":"https://arxiv.org/abs/2010.11967v1","url_pdf":"https://arxiv.org/pdf/2010.11967v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"language-models-are-open-knowledge-graphs-1","repo_url":"https://github.com/cgraywang/deepex","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"language-models-are-open-knowledge-graphs-1","repo_url":"https://github.com/theblackcat102/language-models-are-knowledge-graphs-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.11967","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.11967"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/theblackcat102/language-models-are-knowledge-graphs-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cgraywang/deepex","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"ran_violates":1,"unverified":6},"by_repo_kind":{"listed":{"samples":7,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"d723fb6f7a24f893","entry":"process_matrix","repo":"theblackcat102/language-models-are-knowledge-graphs-pytorch","repo_kind":"listed","path":"process.py","file_url":"https://github.com/theblackcat102/language-models-are-knowledge-graphs-pytorch/blob/HEAD/process.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d723fb6f7a24f893"}},{"code_sha256_prefix":"7c508037b40522af","entry":"str2bool","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"7c508037b40522af"}},{"code_sha256_prefix":"063309af25d9640a","entry":"File","repo":"cgraywang/deepex","repo_kind":"listed","path":"src/deepex/utils.py","file_url":"https://github.com/cgraywang/deepex/blob/HEAD/src/deepex/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"063309af25d9640a"}},{"code_sha256_prefix":"86dd1e534a007846","entry":"Folder","repo":"cgraywang/deepex","repo_kind":"listed","path":"src/deepex/utils.py","file_url":"https://github.com/cgraywang/deepex/blob/HEAD/src/deepex/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86dd1e534a007846"}},{"code_sha256_prefix":"157711cac4c4bfbb","entry":"Prefix","repo":"cgraywang/deepex","repo_kind":"listed","path":"src/deepex/utils.py","file_url":"https://github.com/cgraywang/deepex/blob/HEAD/src/deepex/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"157711cac4c4bfbb"}},{"code_sha256_prefix":"ad6eeb770c5c82b6","entry":"convert_tokens_to_string","repo":"cgraywang/deepex","repo_kind":"listed","path":"src/deepex/model/kgm.py","file_url":"https://github.com/cgraywang/deepex/blob/HEAD/src/deepex/model/kgm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ad6eeb770c5c82b6"}},{"code_sha256_prefix":"54be3ac4fc221899","entry":"layer_attention","repo":"cgraywang/deepex","repo_kind":"listed","path":"src/deepex/model/kgm.py","file_url":"https://github.com/cgraywang/deepex/blob/HEAD/src/deepex/model/kgm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54be3ac4fc221899"}},{"code_sha256_prefix":"2c118e004049dabb","entry":"transform_layer_attention","repo":"cgraywang/deepex","repo_kind":"listed","path":"src/deepex/model/kgm.py","file_url":"https://github.com/cgraywang/deepex/blob/HEAD/src/deepex/model/kgm.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2c118e004049dabb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}