{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/graph-language-models","title":"Graph Language Models","arxiv_id":"2401.07105","date":"2024-01-13","proceeding":null,"authors":["Moritz Plenz","Anette Frank"],"abstract":"While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs -- which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure -- but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility.","url_abs":"https://arxiv.org/abs/2401.07105v3","url_pdf":"https://arxiv.org/pdf/2401.07105v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"graph-language-models","repo_url":"https://github.com/heidelberg-nlp/graphlanguagemodels","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"knowledge-graphs","task_name":"Knowledge Graphs"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"relation-classification","task_name":"Relation Classification"}],"methods":[{"method_slug":"glm","method_name":"GLM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.07105","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.07105"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/heidelberg-nlp/graphlanguagemodels","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"46d1b50d433ced28","entry":"get_all_relations","repo":"heidelberg-nlp/graphlanguagemodels","repo_kind":"official","path":"preprocessing/rebel.py","file_url":"https://github.com/heidelberg-nlp/graphlanguagemodels/blob/HEAD/preprocessing/rebel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46d1b50d433ced28"}},{"code_sha256_prefix":"ac0f301554dd37ee","entry":"get_embedding","repo":"heidelberg-nlp/graphlanguagemodels","repo_kind":"official","path":"preprocessing/relation_prediction_dataset_to_torch_geometric.py","file_url":"https://github.com/heidelberg-nlp/graphlanguagemodels/blob/HEAD/preprocessing/relation_prediction_dataset_to_torch_geometric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac0f301554dd37ee"}},{"code_sha256_prefix":"6d493322a3184c0f","entry":"get_relation_counts","repo":"heidelberg-nlp/graphlanguagemodels","repo_kind":"official","path":"preprocessing/rebel.py","file_url":"https://github.com/heidelberg-nlp/graphlanguagemodels/blob/HEAD/preprocessing/rebel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d493322a3184c0f"}},{"code_sha256_prefix":"169bb21ca5251842","entry":"graph_to_list","repo":"heidelberg-nlp/graphlanguagemodels","repo_kind":"official","path":"preprocessing/relation_prediction_dataset_construction.py","file_url":"https://github.com/heidelberg-nlp/graphlanguagemodels/blob/HEAD/preprocessing/relation_prediction_dataset_construction.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"169bb21ca5251842"}},{"code_sha256_prefix":"5acb41c3f5cc57f3","entry":"load_jsonl","repo":"heidelberg-nlp/graphlanguagemodels","repo_kind":"official","path":"preprocessing/rebel.py","file_url":"https://github.com/heidelberg-nlp/graphlanguagemodels/blob/HEAD/preprocessing/rebel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5acb41c3f5cc57f3"}},{"code_sha256_prefix":"9ba5d35b43a17ef8","entry":"r2nl","repo":"heidelberg-nlp/graphlanguagemodels","repo_kind":"official","path":"models/graph_T5/wrapper_functions.py","file_url":"https://github.com/heidelberg-nlp/graphlanguagemodels/blob/HEAD/models/graph_T5/wrapper_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ba5d35b43a17ef8"}},{"code_sha256_prefix":"d0e5a7cb9ec518fa","entry":"sample_triplets","repo":"heidelberg-nlp/graphlanguagemodels","repo_kind":"official","path":"preprocessing/relation_prediction_dataset_construction.py","file_url":"https://github.com/heidelberg-nlp/graphlanguagemodels/blob/HEAD/preprocessing/relation_prediction_dataset_construction.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0e5a7cb9ec518fa"}},{"code_sha256_prefix":"40931b151c7e02f4","entry":"get_dummy_graph","repo":"heidelberg-nlp/graphlanguagemodels","repo_kind":"official","path":"models/graph_T5/wrapper_functions.py","file_url":"https://github.com/heidelberg-nlp/graphlanguagemodels/blob/HEAD/models/graph_T5/wrapper_functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40931b151c7e02f4"}},{"code_sha256_prefix":"7112073cf1646198","entry":"load_tf_weights_in_t5","repo":"heidelberg-nlp/graphlanguagemodels","repo_kind":"official","path":"models/graph_T5/graph_t5/modeling_t5.py","file_url":"https://github.com/heidelberg-nlp/graphlanguagemodels/blob/HEAD/models/graph_T5/graph_t5/modeling_t5.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7112073cf1646198"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}