{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/colake-contextualized-language-and-knowledge","title":"CoLAKE: Contextualized Language and Knowledge Embedding","arxiv_id":"2010.00309","date":"2020-10-01","proceeding":"COLING 2020 8","authors":["Tianxiang Sun","Yunfan Shao","Xipeng Qiu","Qipeng Guo","Yaru Hu","Xuanjing Huang","Zheng Zhang"],"abstract":"With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the performance gains of these models. Few works explore the potential of deep contextualized knowledge representation when injecting knowledge. In this paper, we propose the Contextualized Language and Knowledge Embedding (CoLAKE), which jointly learns contextualized representation for both language and knowledge with the extended MLM objective. Instead of injecting only entity embeddings, CoLAKE extracts the knowledge context of an entity from large-scale knowledge bases. To handle the heterogeneity of knowledge context and language context, we integrate them in a unified data structure, word-knowledge graph (WK graph). CoLAKE is pre-trained on large-scale WK graphs with the modified Transformer encoder. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks. Experimental results show that CoLAKE outperforms previous counterparts on most of the tasks. Besides, CoLAKE achieves surprisingly high performance on our synthetic task called word-knowledge graph completion, which shows the superiority of simultaneously contextualizing language and knowledge representation.","url_abs":"https://arxiv.org/abs/2010.00309v1","url_pdf":"https://arxiv.org/pdf/2010.00309v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"colake-contextualized-language-and-knowledge","repo_url":"https://github.com/txsun1997/CoLAKE","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"entity-embeddings","task_name":"Entity Embeddings"},{"task_slug":"knowledge-graph-completion","task_name":"Knowledge Graph Completion"},{"task_slug":"knowledge-probing","task_name":"Knowledge Probing"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2010.00309","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2010.00309"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/txsun1997/CoLAKE","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"11b8618df0ea9846","entry":"batchify","repo":"txsun1997/CoLAKE","repo_kind":"official","path":"lama/batch_eval_KB_completion.py","file_url":"https://github.com/txsun1997/CoLAKE/blob/HEAD/lama/batch_eval_KB_completion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"11b8618df0ea9846"}},{"code_sha256_prefix":"8dbb7d15478e3d54","entry":"build_label_vocab","repo":"txsun1997/CoLAKE","repo_kind":"official","path":"finetune/utils.py","file_url":"https://github.com/txsun1997/CoLAKE/blob/HEAD/finetune/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8dbb7d15478e3d54"}},{"code_sha256_prefix":"4319c3972840dc94","entry":"build_temp_ent_vocab","repo":"txsun1997/CoLAKE","repo_kind":"official","path":"finetune/utils.py","file_url":"https://github.com/txsun1997/CoLAKE/blob/HEAD/finetune/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4319c3972840dc94"}},{"code_sha256_prefix":"c23741b45f0d0d67","entry":"download_file_from_google_drive","repo":"txsun1997/CoLAKE","repo_kind":"official","path":"download_gdrive.py","file_url":"https://github.com/txsun1997/CoLAKE/blob/HEAD/download_gdrive.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c23741b45f0d0d67"}},{"code_sha256_prefix":"a2861d9c873073be","entry":"filter_samples","repo":"txsun1997/CoLAKE","repo_kind":"official","path":"lama/batch_eval_KB_completion.py","file_url":"https://github.com/txsun1997/CoLAKE/blob/HEAD/lama/batch_eval_KB_completion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a2861d9c873073be"}},{"code_sha256_prefix":"f553046649ddd6e7","entry":"get_negation_metric","repo":"txsun1997/CoLAKE","repo_kind":"official","path":"lama/evaluation_metrics.py","file_url":"https://github.com/txsun1997/CoLAKE/blob/HEAD/lama/evaluation_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f553046649ddd6e7"}},{"code_sha256_prefix":"c992a67a3fe6661d","entry":"get_ranking","repo":"txsun1997/CoLAKE","repo_kind":"official","path":"lama/evaluation_metrics.py","file_url":"https://github.com/txsun1997/CoLAKE/blob/HEAD/lama/evaluation_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c992a67a3fe6661d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}