{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/graphformers-gnn-nested-language-models-for","title":"GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph","arxiv_id":"2105.02605","date":"2021-05-06","proceeding":"NeurIPS 2021 12","authors":["Junhan Yang","Zheng Liu","Shitao Xiao","Chaozhuo Li","Defu Lian","Sanjay Agrawal","Amit Singh","Guangzhong Sun","Xing Xie"],"abstract":"The representation learning on textual graph is to generate low-dimensional embeddings for the nodes based on the individual textual features and the neighbourhood information. Recent breakthroughs on pretrained language models and graph neural networks push forward the development of corresponding techniques. The existing works mainly rely on the cascaded model architecture: the textual features of nodes are independently encoded by language models at first; the textual embeddings are aggregated by graph neural networks afterwards. However, the above architecture is limited due to the independent modeling of textual features. In this work, we propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow, {making} each node's semantic accurately comprehended from the global perspective. In addition, a {progressive} learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph. Extensive evaluations are conducted on three large-scale benchmark datasets, where GraphFormers outperform the SOTA baselines with comparable running efficiency.","url_abs":"https://arxiv.org/abs/2105.02605v3","url_pdf":"https://arxiv.org/pdf/2105.02605v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"graphformers-gnn-nested-language-models-for","repo_url":"https://github.com/microsoft/graphformers","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"recommendation-systems","task_name":"Recommendation Systems"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.02605","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.02605"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/GraphFormers","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/graphformers","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_honours":2,"ran_violates":1,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/models/tnlrv3/modeling_decoding.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/models/tnlrv3/modeling_decoding.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"5f6cca96082d0328","entry":"acc","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/utils.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f6cca96082d0328"}},{"code_sha256_prefix":"43e79e8e0904d5d8","entry":"dcg_score","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"43e79e8e0904d5d8"}},{"code_sha256_prefix":"40e9fee2e0b7e278","entry":"gelu","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/models/tnlrv3/modeling_decoding.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/models/tnlrv3/modeling_decoding.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"40e9fee2e0b7e278"}},{"code_sha256_prefix":"7c508037b40522af","entry":"str2bool","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c508037b40522af"}},{"code_sha256_prefix":"0185bc550948bb9e","entry":"batch_list_to_batch_tensors","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/models/tnlrv3/s2s_loader.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/models/tnlrv3/s2s_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0185bc550948bb9e"}},{"code_sha256_prefix":"3c1f5f7bfb4716fa","entry":"create_mask_and_position_ids","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/models/tnlrv3/modeling.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/models/tnlrv3/modeling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c1f5f7bfb4716fa"}},{"code_sha256_prefix":"8321d846789e75f1","entry":"get_random_word","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/models/tnlrv3/s2s_loader.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/models/tnlrv3/s2s_loader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8321d846789e75f1"}},{"code_sha256_prefix":"91d5980aeafe04e4","entry":"load_model","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/models/tnlrv3/convert_state_dict.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/models/tnlrv3/convert_state_dict.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91d5980aeafe04e4"}},{"code_sha256_prefix":"0fa0e6d25bc5b48e","entry":"relative_position_bucket","repo":"microsoft/GraphFormers","repo_kind":"official","path":"src/models/tnlrv3/modeling.py","file_url":"https://github.com/microsoft/GraphFormers/blob/HEAD/src/models/tnlrv3/modeling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0fa0e6d25bc5b48e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}