{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-bidirectional-language-knowledge-graph","title":"Deep Bidirectional Language-Knowledge Graph Pretraining","arxiv_id":"2210.09338","date":"2022-10-17","proceeding":null,"authors":["Michihiro Yasunaga","Antoine Bosselut","Hongyu Ren","Xikun Zhang","Christopher D Manning","Percy Liang","Jure Leskovec"],"abstract":"Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves notable performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our code and trained models are available at https://github.com/michiyasunaga/dragon.","url_abs":"https://arxiv.org/abs/2210.09338v2","url_pdf":"https://arxiv.org/pdf/2210.09338v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-bidirectional-language-knowledge-graph","repo_url":"https://github.com/michiyasunaga/dragon","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"deep-bidirectional-language-knowledge-graph","repo_url":"https://github.com/HaochenLiu2000/QAP","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"common-sense-reasoning","task_name":"Common Sense Reasoning"},{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"link-prediction","task_name":"Link Prediction"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"riddle-sense","task_name":"Riddle Sense"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/common-sense-reasoning-on-commonsenseqa","task":"Common Sense Reasoning","dataset":"CommonsenseQA","model":"DRAGON","rank_in_archive_order":9,"of":38,"metrics":{"Accuracy":"78.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-medqa-usmle","task":"Question Answering","dataset":"MedQA","model":"DRAGON + BioLinkBERT","rank_in_archive_order":18,"of":27,"metrics":{"Accuracy":"47.5"},"uses_additional_data":false},{"leaderboard":"/sota/riddle-sense-on-riddle-sense","task":"Riddle Sense","dataset":"RiddleSense","model":"DRAGON","rank_in_archive_order":1,"of":3,"metrics":{"Accuracy (%)":"71.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.09338","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.09338"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/michiyasunaga/dragon","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HaochenLiu2000/QAP","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":1,"unverified":16},"by_repo_kind":{"official":{"samples":18,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b75e9c793d2a286a","entry":"gelu","repo":"michiyasunaga/dragon","repo_kind":"official","path":"utils/layers.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/utils/layers.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b75e9c793d2a286a"}},{"code_sha256_prefix":"b7b66139532f1ae7","entry":"sep_params","repo":"michiyasunaga/dragon","repo_kind":"official","path":"dragon.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/dragon.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7b66139532f1ae7"}},{"code_sha256_prefix":"b256ca817b4ee5e6","entry":"clean_json_line","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/convert_csqa.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/convert_csqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b256ca817b4ee5e6"}},{"code_sha256_prefix":"90b1b740bfed326c","entry":"concepts2adj","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/graph.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/graph.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"90b1b740bfed326c"}},{"code_sha256_prefix":"be8c35c9c1d52b66","entry":"construct_graph","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/conceptnet.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/conceptnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be8c35c9c1d52b66"}},{"code_sha256_prefix":"15baaf34a0e42ec6","entry":"convert_qajson_to_entailment","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/convert_csqa.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/convert_csqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"15baaf34a0e42ec6"}},{"code_sha256_prefix":"174ae40015de395f","entry":"convert_qajson_to_entailment","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/convert_obqa.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/convert_obqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"174ae40015de395f"}},{"code_sha256_prefix":"345456d568545c8e","entry":"create_output_dict","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/convert_obqa.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/convert_obqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"345456d568545c8e"}},{"code_sha256_prefix":"5f4eefd99c11e698","entry":"del_pos","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/conceptnet.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/conceptnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f4eefd99c11e698"}},{"code_sha256_prefix":"866c784c24ba2fa8","entry":"dropout_mask","repo":"michiyasunaga/dragon","repo_kind":"official","path":"utils/layers.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/utils/layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"866c784c24ba2fa8"}},{"code_sha256_prefix":"96ec8d520ae857f1","entry":"generate_adj_matrix_per_inst","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/graph.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/graph.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96ec8d520ae857f1"}},{"code_sha256_prefix":"a9a454bf4414e3fe","entry":"get_fitb_from_question","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/convert_csqa.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/convert_csqa.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a9a454bf4414e3fe"}},{"code_sha256_prefix":"29a699b766ece561","entry":"get_glove_score","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/graph_umls_with_glove.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/graph_umls_with_glove.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29a699b766ece561"}},{"code_sha256_prefix":"9dcf00b849de8977","entry":"glove_init","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/conceptnet.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/conceptnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9dcf00b849de8977"}},{"code_sha256_prefix":"440036c58abd9ccc","entry":"make_one_hot","repo":"michiyasunaga/dragon","repo_kind":"official","path":"modeling/modeling_gnn.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/modeling/modeling_gnn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"440036c58abd9ccc"}},{"code_sha256_prefix":"3841430a72ff67fa","entry":"masked_softmax","repo":"michiyasunaga/dragon","repo_kind":"official","path":"utils/layers.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/utils/layers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3841430a72ff67fa"}},{"code_sha256_prefix":"66135965fe362f47","entry":"plain_graph_generation","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/graph.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/graph.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"66135965fe362f47"}},{"code_sha256_prefix":"fe81b86935035b80","entry":"sent2glove","repo":"michiyasunaga/dragon","repo_kind":"official","path":"preprocess_utils/graph_umls_with_glove.py","file_url":"https://github.com/michiyasunaga/dragon/blob/HEAD/preprocess_utils/graph_umls_with_glove.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe81b86935035b80"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}