{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kelm-knowledge-enhanced-pre-trained-language","title":"KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs","arxiv_id":"2109.04223","date":"2021-09-09","proceeding":null,"authors":["Yinquan Lu","Haonan Lu","Guirong Fu","Qun Liu"],"abstract":"Incorporating factual knowledge into pre-trained language models (PLM) such as BERT is an emerging trend in recent NLP studies. However, most of the existing methods combine the external knowledge integration module with a modified pre-training loss and re-implement the pre-training process on the large-scale corpus. Re-pretraining these models is usually resource-consuming, and difficult to adapt to another domain with a different knowledge graph (KG). Besides, those works either cannot embed knowledge context dynamically according to textual context or struggle with the knowledge ambiguity issue. In this paper, we propose a novel knowledge-aware language model framework based on fine-tuning process, which equips PLM with a unified knowledge-enhanced text graph that contains both text and multi-relational sub-graphs extracted from KG. We design a hierarchical relational-graph-based message passing mechanism, which can allow the representations of injected KG and text to mutually update each other and can dynamically select ambiguous mentioned entities that share the same text. Our empirical results show that our model can efficiently incorporate world knowledge from KGs into existing language models such as BERT, and achieve significant improvement on the machine reading comprehension (MRC) task compared with other knowledge-enhanced models.","url_abs":"https://arxiv.org/abs/2109.04223v2","url_pdf":"https://arxiv.org/pdf/2109.04223v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kelm-knowledge-enhanced-pre-trained-language","repo_url":"https://github.com/nlp-anonymous-happy/anonymous-kg-guided-nlp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-reading-comprehension","task_name":"Machine Reading Comprehension"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"KELM (finetuning RoBERTa-large based single model)","rank_in_archive_order":11,"of":45,"metrics":{"EM":"89.1","F1":"89.6"},"uses_additional_data":false},{"leaderboard":"/sota/common-sense-reasoning-on-record","task":"Common Sense Reasoning","dataset":"ReCoRD","model":"KELM (finetuning BERT-large based single model)","rank_in_archive_order":21,"of":45,"metrics":{"EM":"76.2","F1":"76.7"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-copa","task":"Question Answering","dataset":"COPA","model":"KELM (finetuning BERT-large based single model)","rank_in_archive_order":39,"of":60,"metrics":{"Accuracy":"78.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-multirc","task":"Question Answering","dataset":"MultiRC","model":"KELM (finetuning BERT-large based single model)","rank_in_archive_order":15,"of":30,"metrics":{"EM":"27.2","F1":"70.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2109.04223","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.04223"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nlp-anonymous-happy/anonymous-kg-guided-nlp","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9df40357afea56cc","entry":"to_list","repo":"nlp-anonymous-happy/anonymous-kg-guided-nlp","repo_kind":"official","path":"src/run_copa.py","file_url":"https://github.com/nlp-anonymous-happy/anonymous-kg-guided-nlp/blob/HEAD/src/run_copa.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9df40357afea56cc"}},{"code_sha256_prefix":"881ea8793bf870d1","entry":"configure_tokenizer_model_bert","repo":"nlp-anonymous-happy/anonymous-kg-guided-nlp","repo_kind":"official","path":"src/model/bert.py","file_url":"https://github.com/nlp-anonymous-happy/anonymous-kg-guided-nlp/blob/HEAD/src/model/bert.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"881ea8793bf870d1"}},{"code_sha256_prefix":"88f0e09ed13a1e65","entry":"preprocess_nell_ent_name","repo":"nlp-anonymous-happy/anonymous-kg-guided-nlp","repo_kind":"official","path":"src/kgs_retrieve/nell.py","file_url":"https://github.com/nlp-anonymous-happy/anonymous-kg-guided-nlp/blob/HEAD/src/kgs_retrieve/nell.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88f0e09ed13a1e65"}},{"code_sha256_prefix":"2ea3a5a500562286","entry":"preprocess_record_ent_name","repo":"nlp-anonymous-happy/anonymous-kg-guided-nlp","repo_kind":"official","path":"src/kgs_retrieve/nell.py","file_url":"https://github.com/nlp-anonymous-happy/anonymous-kg-guided-nlp/blob/HEAD/src/kgs_retrieve/nell.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2ea3a5a500562286"}},{"code_sha256_prefix":"e97ed6b90bab1416","entry":"read_concept_embedding","repo":"nlp-anonymous-happy/anonymous-kg-guided-nlp","repo_kind":"official","path":"src/kgs_retrieve/baseretriever.py","file_url":"https://github.com/nlp-anonymous-happy/anonymous-kg-guided-nlp/blob/HEAD/src/kgs_retrieve/baseretriever.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e97ed6b90bab1416"}},{"code_sha256_prefix":"5efae5d998e8fd45","entry":"run_strip_accents","repo":"nlp-anonymous-happy/anonymous-kg-guided-nlp","repo_kind":"official","path":"src/kgs_retrieve/baseretriever.py","file_url":"https://github.com/nlp-anonymous-happy/anonymous-kg-guided-nlp/blob/HEAD/src/kgs_retrieve/baseretriever.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5efae5d998e8fd45"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}