{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/investigating-expressiveness-of-transformer","title":"How Expressive are Transformers in Spectral Domain for Graphs?","arxiv_id":"2201.09332","date":"2022-01-23","proceeding":null,"authors":["Anson Bastos","Abhishek Nadgeri","Kuldeep Singh","Hiroki Kanezashi","Toyotaro Suzumura","Isaiah Onando Mulang'"],"abstract":"The recent works proposing transformer-based models for graphs have proven the inadequacy of Vanilla Transformer for graph representation learning. To understand this inadequacy, there is a need to investigate if spectral analysis of the transformer will reveal insights into its expressive power. Similar studies already established that spectral analysis of Graph neural networks (GNNs) provides extra perspectives on their expressiveness. In this work, we systematically study and establish the link between the spatial and spectral domain in the realm of the transformer. We further provide a theoretical analysis and prove that the spatial attention mechanism in the transformer cannot effectively capture the desired frequency response, thus, inherently limiting its expressiveness in spectral space. Therefore, we propose FeTA, a framework that aims to perform attention over the entire graph spectrum (i.e., actual frequency components of the graphs) analogous to the attention in spatial space. Empirical results suggest that FeTA provides homogeneous performance gain against vanilla transformer across all tasks on standard benchmarks and can easily be extended to GNN-based models with low-pass characteristics (e.g., GAT).","url_abs":"https://arxiv.org/abs/2201.09332v4","url_pdf":"https://arxiv.org/pdf/2201.09332v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"investigating-expressiveness-of-transformer","repo_url":"https://github.com/ansonb/FeTA_TMLR","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"graph-representation-learning","task_name":"Graph Representation Learning"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2201.09332","atlas_url":"https://app.syntology.ai/?focus=2201.09332","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2201.09332"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ansonb/FeTA_TMLR","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1b26e6ea6d2db1d1","entry":"cheb_msg","repo":"ansonb/FeTA_TMLR","repo_kind":"official","path":"LPE/layers/gat_feta_layer.py","file_url":"https://github.com/ansonb/FeTA_TMLR/blob/HEAD/LPE/layers/gat_feta_layer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b26e6ea6d2db1d1"}},{"code_sha256_prefix":"85162c58be024fa2","entry":"gcn_msg","repo":"ansonb/FeTA_TMLR","repo_kind":"official","path":"LPE/layers/gat_feta_layer.py","file_url":"https://github.com/ansonb/FeTA_TMLR/blob/HEAD/LPE/layers/gat_feta_layer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"85162c58be024fa2"}},{"code_sha256_prefix":"700de849477f93ce","entry":"imp_exp_attn","repo":"ansonb/FeTA_TMLR","repo_kind":"official","path":"LPE/layers/GT_layer_bkp.py","file_url":"https://github.com/ansonb/FeTA_TMLR/blob/HEAD/LPE/layers/GT_layer_bkp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"700de849477f93ce"}},{"code_sha256_prefix":"a3fb87a259b15eae","entry":"scaling","repo":"ansonb/FeTA_TMLR","repo_kind":"official","path":"LPE/layers/GT_layer_bkp.py","file_url":"https://github.com/ansonb/FeTA_TMLR/blob/HEAD/LPE/layers/GT_layer_bkp.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3fb87a259b15eae"}},{"code_sha256_prefix":"7e094af0e4a3779e","entry":"send_source","repo":"ansonb/FeTA_TMLR","repo_kind":"official","path":"LPE/layers/gat_feta_layer.py","file_url":"https://github.com/ansonb/FeTA_TMLR/blob/HEAD/LPE/layers/gat_feta_layer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e094af0e4a3779e"}},{"code_sha256_prefix":"9408c2ca6e799410","entry":"src_dot_dst","repo":"ansonb/FeTA_TMLR","repo_kind":"official","path":"LPE/layers/GT_layer_bkp.py","file_url":"https://github.com/ansonb/FeTA_TMLR/blob/HEAD/LPE/layers/GT_layer_bkp.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9408c2ca6e799410"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}