{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/less-is-more-on-the-over-globalizing-problem","title":"Less is More: on the Over-Globalizing Problem in Graph Transformers","arxiv_id":"2405.01102","date":"2024-05-02","proceeding":null,"authors":["Yujie Xing","Xiao Wang","Yibo Li","Hai Huang","Chuan Shi"],"abstract":"Graph Transformer, due to its global attention mechanism, has emerged as a new tool in dealing with graph-structured data. It is well recognized that the global attention mechanism considers a wider receptive field in a fully connected graph, leading many to believe that useful information can be extracted from all the nodes. In this paper, we challenge this belief: does the globalizing property always benefit Graph Transformers? We reveal the over-globalizing problem in Graph Transformer by presenting both empirical evidence and theoretical analysis, i.e., the current attention mechanism overly focuses on those distant nodes, while the near nodes, which actually contain most of the useful information, are relatively weakened. Then we propose a novel Bi-Level Global Graph Transformer with Collaborative Training (CoBFormer), including the inter-cluster and intra-cluster Transformers, to prevent the over-globalizing problem while keeping the ability to extract valuable information from distant nodes. Moreover, the collaborative training is proposed to improve the model's generalization ability with a theoretical guarantee. Extensive experiments on various graphs well validate the effectiveness of our proposed CoBFormer.","url_abs":"https://arxiv.org/abs/2405.01102v2","url_pdf":"https://arxiv.org/pdf/2405.01102v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"less-is-more-on-the-over-globalizing-problem","repo_url":"https://github.com/null-xyj/CoBFormer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"graph-transformer","method_name":"Graph Transformer"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"lapeigen","method_name":"LapEigen"},{"method_slug":"laplacian-pe","method_name":"Laplacian PE"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2405.01102","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.01102"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/null-xyj/CoBFormer","reach":null}],"summary":{"ran":3,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"540ee9a6b0ddd843","entry":"FFN","repo":"null-xyj/CoBFormer","repo_kind":"official","path":"Model/CoBFormer.py","file_url":"https://github.com/null-xyj/CoBFormer/blob/HEAD/Model/CoBFormer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"540ee9a6b0ddd843"}},{"code_sha256_prefix":"bd341cc57b38b439","entry":"MultiHeadAttention","repo":"null-xyj/CoBFormer","repo_kind":"official","path":"Model/CoBFormer.py","file_url":"https://github.com/null-xyj/CoBFormer/blob/HEAD/Model/CoBFormer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd341cc57b38b439"}},{"code_sha256_prefix":"7412a7321f21cb39","entry":"ScaledDotProductAttention","repo":"null-xyj/CoBFormer","repo_kind":"official","path":"Model/CoBFormer.py","file_url":"https://github.com/null-xyj/CoBFormer/blob/HEAD/Model/CoBFormer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7412a7321f21cb39"}},{"code_sha256_prefix":"89e1bf61d9afa1f1","entry":"BGA","repo":"null-xyj/CoBFormer","repo_kind":"official","path":"Model/CoBFormer.py","file_url":"https://github.com/null-xyj/CoBFormer/blob/HEAD/Model/CoBFormer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"89e1bf61d9afa1f1"}},{"code_sha256_prefix":"1326b689688c1189","entry":"BGALayer","repo":"null-xyj/CoBFormer","repo_kind":"official","path":"Model/CoBFormer.py","file_url":"https://github.com/null-xyj/CoBFormer/blob/HEAD/Model/CoBFormer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1326b689688c1189"}},{"code_sha256_prefix":"39445b2bd89ee3e9","entry":"CoBFormer","repo":"null-xyj/CoBFormer","repo_kind":"official","path":"Model/CoBFormer.py","file_url":"https://github.com/null-xyj/CoBFormer/blob/HEAD/Model/CoBFormer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39445b2bd89ee3e9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}