{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/from-graph-low-rank-global-attention-to-2-fwl","title":"Global Attention Improves Graph Networks Generalization","arxiv_id":"2006.07846","date":"2020-06-14","proceeding":null,"authors":["Omri Puny","Heli Ben-Hamu","Yaron Lipman"],"abstract":"This paper advocates incorporating a Low-Rank Global Attention (LRGA) module, a computation and memory efficient variant of the dot-product attention (Vaswani et al., 2017), to Graph Neural Networks (GNNs) for improving their generalization power. To theoretically quantify the generalization properties granted by adding the LRGA module to GNNs, we focus on a specific family of expressive GNNs and show that augmenting it with LRGA provides algorithmic alignment to a powerful graph isomorphism test, namely the 2-Folklore Weisfeiler-Lehman (2-FWL) algorithm. In more detail we: (i) consider the recent Random Graph Neural Network (RGNN) (Sato et al., 2020) framework and prove that it is universal in probability; (ii) show that RGNN augmented with LRGA aligns with 2-FWL update step via polynomial kernels; and (iii) bound the sample complexity of the kernel's feature map when learned with a randomly initialized two-layer MLP. From a practical point of view, augmenting existing GNN layers with LRGA produces state of the art results in current GNN benchmarks. Lastly, we observe that augmenting various GNN architectures with LRGA often closes the performance gap between different models.","url_abs":"https://arxiv.org/abs/2006.07846v2","url_pdf":"https://arxiv.org/pdf/2006.07846v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"from-graph-low-rank-global-attention-to-2-fwl","repo_url":"https://github.com/chuanqichen/cs224w","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"from-graph-low-rank-global-attention-to-2-fwl","repo_url":"https://github.com/omri1348/LRGA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"from-graph-low-rank-global-attention-to-2-fwl","repo_url":"https://github.com/omri1348/LRGA/tree/master/ogb/examples/linkproppred","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"graph-neural-network","task_name":"Graph Neural Network"}],"methods":[{"method_slug":"graph-neural-network","method_name":"Graph Neural Network"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/link-property-prediction-on-ogbl-collab","task":"Link Property Prediction","dataset":"ogbl-collab","model":"PLNLP+ LRGA","rank_in_archive_order":7,"of":34,"metrics":{"Ext. data":"No","Number of params":"35200656","Test Hits@50":"0.6909 ± 0.0055","Validation Hits@50":"1.0000 ± 0.0000"},"uses_additional_data":false},{"leaderboard":"/sota/link-property-prediction-on-ogbl-collab","task":"Link Property Prediction","dataset":"ogbl-collab","model":"LRGA + GCN","rank_in_archive_order":23,"of":34,"metrics":{"Ext. data":"No","Number of params":"1069489","Test Hits@50":"0.5221 ± 0.0072","Validation Hits@50":"0.6088 ± 0.0059"},"uses_additional_data":false},{"leaderboard":"/sota/link-property-prediction-on-ogbl-ddi","task":"Link Property Prediction","dataset":"ogbl-ddi","model":"LRGA + GCN","rank_in_archive_order":20,"of":31,"metrics":{"Ext. data":"No","Number of params":"1576081","Test Hits@20":"0.6230 ± 0.0912","Validation Hits@20":"0.6675 ± 0.0058"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2006.07846","atlas_url":"https://app.syntology.ai/?focus=2006.07846","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.07846"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/omri1348/LRGA/tree/master/ogb/examples/linkproppred","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chuanqichen/cs224w","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/omri1348/LRGA","reach":null}],"summary":{"ran_draft_wrong":2,"ran_fixture":1,"ran_honours":1},"by_repo_kind":{"listed":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"01da28167d92f23a","entry":"gpu_setup","repo":"omri1348/LRGA","repo_kind":"listed","path":"ogb/examples/linkproppred/collab/gnn.py","file_url":"https://github.com/omri1348/LRGA/blob/HEAD/ogb/examples/linkproppred/collab/gnn.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01da28167d92f23a"}},{"code_sha256_prefix":"7d01f508ca84b432","entry":"joint_normalize2","repo":"omri1348/LRGA","repo_kind":"listed","path":"ogb/examples/linkproppred/collab/graph_global_attention_layer.py","file_url":"https://github.com/omri1348/LRGA/blob/HEAD/ogb/examples/linkproppred/collab/graph_global_attention_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d01f508ca84b432"}},{"code_sha256_prefix":"aed917122fd597e7","entry":"test","repo":"omri1348/LRGA","repo_kind":"listed","path":"ogb/examples/linkproppred/collab/gnn.py","file_url":"https://github.com/omri1348/LRGA/blob/HEAD/ogb/examples/linkproppred/collab/gnn.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aed917122fd597e7"}},{"code_sha256_prefix":"a2328b5e3445a490","entry":"train","repo":"omri1348/LRGA","repo_kind":"listed","path":"ogb/examples/linkproppred/collab/gnn.py","file_url":"https://github.com/omri1348/LRGA/blob/HEAD/ogb/examples/linkproppred/collab/gnn.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a2328b5e3445a490"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}