{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gradformer-graph-transformer-with-exponential","title":"Gradformer: Graph Transformer with Exponential Decay","arxiv_id":"2404.15729","date":"2024-04-24","proceeding":null,"authors":["Chuang Liu","Zelin Yao","Yibing Zhan","Xueqi Ma","Shirui Pan","Wenbin Hu"],"abstract":"Graph Transformers (GTs) have demonstrated their advantages across a wide range of tasks. However, the self-attention mechanism in GTs overlooks the graph's inductive biases, particularly biases related to structure, which are crucial for the graph tasks. Although some methods utilize positional encoding and attention bias to model inductive biases, their effectiveness is still suboptimal analytically. Therefore, this paper presents Gradformer, a method innovatively integrating GT with the intrinsic inductive bias by applying an exponential decay mask to the attention matrix. Specifically, the values in the decay mask matrix diminish exponentially, correlating with the decreasing node proximities within the graph structure. This design enables Gradformer to retain its ability to capture information from distant nodes while focusing on the graph's local details. Furthermore, Gradformer introduces a learnable constraint into the decay mask, allowing different attention heads to learn distinct decay masks. Such an design diversifies the attention heads, enabling a more effective assimilation of diverse structural information within the graph. Extensive experiments on various benchmarks demonstrate that Gradformer consistently outperforms the Graph Neural Network and GT baseline models in various graph classification and regression tasks. Additionally, Gradformer has proven to be an effective method for training deep GT models, maintaining or even enhancing accuracy compared to shallow models as the network deepens, in contrast to the significant accuracy drop observed in other GT models.Codes are available at \\url{https://github.com/LiuChuang0059/Gradformer}.","url_abs":"https://arxiv.org/abs/2404.15729v1","url_pdf":"https://arxiv.org/pdf/2404.15729v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gradformer-graph-transformer-with-exponential","repo_url":"https://github.com/liuchuang0059/gradformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"graph-classification","task_name":"Graph Classification"},{"task_slug":"graph-neural-network","task_name":"Graph Neural Network"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"}],"methods":[{"method_slug":"exponential-decay","method_name":"Exponential Decay"},{"method_slug":"gts","method_name":"GTS"},{"method_slug":"graph-neural-network","method_name":"Graph Neural Network"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.15729","atlas_url":"https://app.syntology.ai/?focus=2404.15729","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.15729"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liuchuang0059/gradformer","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/LiuChuang0059/Gradformer","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"9c678e872e30ebb8","entry":"accuracy_sbm","repo":"LiuChuang0059/Gradformer","repo_kind":"official","path":"criterion/weight_loss.py","file_url":"https://github.com/LiuChuang0059/Gradformer/blob/HEAD/criterion/weight_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9c678e872e30ebb8"}},{"code_sha256_prefix":"2936d5ee68562846","entry":"attention","repo":"LiuChuang0059/Gradformer","repo_kind":"official","path":"model/Attention.py","file_url":"https://github.com/LiuChuang0059/Gradformer/blob/HEAD/model/Attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2936d5ee68562846"}},{"code_sha256_prefix":"1b1a4e44524606fc","entry":"clones","repo":"LiuChuang0059/Gradformer","repo_kind":"official","path":"model/Attention.py","file_url":"https://github.com/LiuChuang0059/Gradformer/blob/HEAD/model/Attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b1a4e44524606fc"}},{"code_sha256_prefix":"65b113a451720c90","entry":"drop_path","repo":"LiuChuang0059/Gradformer","repo_kind":"official","path":"utils/droppath.py","file_url":"https://github.com/LiuChuang0059/Gradformer/blob/HEAD/utils/droppath.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"65b113a451720c90"}},{"code_sha256_prefix":"f46f264157437aa0","entry":"get_scheduler","repo":"LiuChuang0059/Gradformer","repo_kind":"official","path":"optimizer/ultra_optimizer.py","file_url":"https://github.com/LiuChuang0059/Gradformer/blob/HEAD/optimizer/ultra_optimizer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f46f264157437aa0"}},{"code_sha256_prefix":"205244c3de692745","entry":"weighted_cross_entropy","repo":"LiuChuang0059/Gradformer","repo_kind":"official","path":"criterion/weight_loss.py","file_url":"https://github.com/LiuChuang0059/Gradformer/blob/HEAD/criterion/weight_loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"205244c3de692745"}},{"code_sha256_prefix":"0826732a2a55c72d","entry":"cal_full_pinv","repo":"LiuChuang0059/Gradformer","repo_kind":"official","path":"utils/commute.py","file_url":"https://github.com/LiuChuang0059/Gradformer/blob/HEAD/utils/commute.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0826732a2a55c72d"}},{"code_sha256_prefix":"450ae523bd9c18c9","entry":"cal_pinv","repo":"LiuChuang0059/Gradformer","repo_kind":"official","path":"utils/commute.py","file_url":"https://github.com/LiuChuang0059/Gradformer/blob/HEAD/utils/commute.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"450ae523bd9c18c9"}},{"code_sha256_prefix":"9b87967cea992aa0","entry":"fn","repo":"LiuChuang0059/Gradformer","repo_kind":"official","path":"data_utils/load_data.py","file_url":"https://github.com/LiuChuang0059/Gradformer/blob/HEAD/data_utils/load_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b87967cea992aa0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}