{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-attentive-are-graph-attention-networks","title":"How Attentive are Graph Attention Networks?","arxiv_id":"2105.14491","date":"2021-05-30","proceeding":"ICLR 2022 4","authors":["Shaked Brody","Uri Alon","Eran Yahav"],"abstract":"Graph Attention Networks (GATs) are one of the most popular GNN architectures and are considered as the state-of-the-art architecture for representation learning with graphs. In GAT, every node attends to its neighbors given its own representation as the query. However, in this paper we show that GAT computes a very limited kind of attention: the ranking of the attention scores is unconditioned on the query node. We formally define this restricted kind of attention as static attention and distinguish it from a strictly more expressive dynamic attention. Because GATs use a static attention mechanism, there are simple graph problems that GAT cannot express: in a controlled problem, we show that static attention hinders GAT from even fitting the training data. To remove this limitation, we introduce a simple fix by modifying the order of operations and propose GATv2: a dynamic graph attention variant that is strictly more expressive than GAT. We perform an extensive evaluation and show that GATv2 outperforms GAT across 11 OGB and other benchmarks while we match their parametric costs. Our code is available at https://github.com/tech-srl/how_attentive_are_gats . GATv2 is available as part of the PyTorch Geometric library, the Deep Graph Library, and the TensorFlow GNN library.","url_abs":"https://arxiv.org/abs/2105.14491v3","url_pdf":"https://arxiv.org/pdf/2105.14491v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-attentive-are-graph-attention-networks","repo_url":"https://github.com/tech-srl/how_attentive_are_gats","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"how-attentive-are-graph-attention-networks","repo_url":"https://github.com/briangodwinlim/sir-gcn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"CC-BY-4.0"}},{"paper_slug":"how-attentive-are-graph-attention-networks","repo_url":"https://github.com/machadoprx/GATv2-TensorFlow2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"how-attentive-are-graph-attention-networks","repo_url":"https://github.com/dmlc/dgl/blob/master/python/dgl/nn/pytorch/conv/gatv2conv.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"how-attentive-are-graph-attention-networks","repo_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"how-attentive-are-graph-attention-networks","repo_url":"https://github.com/mahequn001/graphlearning/tree/master/model_zoo/gatv2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"how-attentive-are-graph-attention-networks","repo_url":"https://github.com/rusty1s/pytorch_geometric/blob/master/torch_geometric/nn/conv/gatv2_conv.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"how-attentive-are-graph-attention-networks","repo_url":"https://github.com/tensorflow/gnn/blob/main/tensorflow_gnn/docs/api_docs/python/gnn/keras/layers/GATv2.md","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"graph-attention","task_name":"Graph Attention"},{"task_slug":"graph-classification","task_name":"Graph Classification"},{"task_slug":"graph-property-prediction","task_name":"Graph Property Prediction"},{"task_slug":"graph-regression","task_name":"Graph Regression"},{"task_slug":"link-prediction","task_name":"Link Prediction"},{"task_slug":"molecular-property-prediction","task_name":"Molecular Property Prediction"},{"task_slug":"node-property-prediction","task_name":"Node Property Prediction"},{"task_slug":"representation-learning","task_name":"Representation Learning"}],"methods":[{"method_slug":"gatv2","method_name":"GATv2"}],"datasets_introduced":[],"methods_introduced":[{"slug":"gatv2","name":"GATv2","full_name":"Graph Attention Network v2"}],"results":[{"leaderboard":"/sota/graph-classification-on-dd","task":"Graph Classification","dataset":"D&D","model":"GATv2","rank_in_archive_order":42,"of":53,"metrics":{"Accuracy":"75.966±2.191"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-enzymes","task":"Graph Classification","dataset":"ENZYMES","model":"GATv2","rank_in_archive_order":5,"of":54,"metrics":{"Accuracy":"77.987±2.112"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-imdb-b","task":"Graph Classification","dataset":"IMDb-B","model":"GATv2","rank_in_archive_order":7,"of":51,"metrics":{"Accuracy":"80.000±2.739"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-nci1","task":"Graph Classification","dataset":"NCI1","model":"GATv2","rank_in_archive_order":31,"of":69,"metrics":{"Accuracy":"82.384±1.700"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-nci109","task":"Graph Classification","dataset":"NCI109","model":"GATv2","rank_in_archive_order":14,"of":38,"metrics":{"Accuracy":"83.092±0.764"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-proteins","task":"Graph Classification","dataset":"PROTEINS","model":"GATv2","rank_in_archive_order":28,"of":103,"metrics":{"Accuracy":"77.679±2.187"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-esr2","task":"Graph Regression","dataset":"ESR2","model":"GATv2","rank_in_archive_order":6,"of":9,"metrics":{"R2":"0.655±0.000","RMSE":"0.518±0.655"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-f2","task":"Graph Regression","dataset":"F2","model":"GATv2","rank_in_archive_order":6,"of":9,"metrics":{"R2":"0.885±0.000","RMSE":"0.344±0.885"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-kit","task":"Graph Regression","dataset":"KIT","model":"GATv2","rank_in_archive_order":6,"of":9,"metrics":{"R2":"0.826±0.000","RMSE":"0.453±0.826"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-lipophilicity","task":"Graph Regression","dataset":"Lipophilicity","model":"GATv2","rank_in_archive_order":2,"of":23,"metrics":{"R2":"0.821±0.009","RMSE":"0.534±0.014"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-parp1","task":"Graph Regression","dataset":"PARP1","model":"GATv2","rank_in_archive_order":6,"of":9,"metrics":{"R2":"0.919±0.000","RMSE":"0.356±0.919"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-pgr","task":"Graph Regression","dataset":"PGR","model":"GATv2","rank_in_archive_order":7,"of":9,"metrics":{"R2":"0.666±0.000","RMSE":"0.558±0.666"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-zinc-full","task":"Graph Regression","dataset":"ZINC-full","model":"GATv2","rank_in_archive_order":17,"of":19,"metrics":{"Test MAE":"0.079±0.004"},"uses_additional_data":false},{"leaderboard":"/sota/molecular-property-prediction-on-esol","task":"Molecular Property Prediction","dataset":"ESOL","model":"GATv2","rank_in_archive_order":7,"of":20,"metrics":{"R2":"0.928±0.005","RMSE":"0.549±0.020"},"uses_additional_data":false},{"leaderboard":"/sota/molecular-property-prediction-on-freesolv","task":"Molecular Property Prediction","dataset":"FreeSolv","model":"GATv2","rank_in_archive_order":3,"of":22,"metrics":{"R2":"0.970±0.007","RMSE":"0.676±0.081"},"uses_additional_data":false},{"leaderboard":"/sota/node-property-prediction-on-ogbn-arxiv","task":"Node Property Prediction","dataset":"ogbn-arxiv","model":"GIANT+XRT+GATv2","rank_in_archive_order":26,"of":86,"metrics":{"Ext. data":"Yes","Number of params":"207520","Test Accuracy":"0.7415 ± 0.0005","Validation Accuracy":"0.7527 ± 0.0008"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2105.14491","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2105.14491"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tech-srl/how_attentive_are_","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/tech-srl/how_attentive_are_gats","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mahequn001/graphlearning/tree/master/model_zoo/gatv2","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tensorflow/gnn/blob/main/tensorflow_gnn/docs/api_docs/python/gnn/keras/layers/GATv2.md","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/briangodwinlim/sir-gcn","reach":{"status":"ok","spdx":"CC-BY-4.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/machadoprx/GATv2-TensorFlow2","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dmlc/dgl/blob/master/python/dgl/nn/pytorch/conv/gatv2conv.py","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rusty1s/pytorch_geometric/blob/master/torch_geometric/nn/conv/gatv2_conv.py","reach":null}],"summary":{"ran":6,"unverified":10},"by_repo_kind":{"listed":{"samples":16,"ran":6,"repositories":5}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"365d918744e8f7d5","entry":"GATv2","repo":"labmlai/annotated_deep_learning_paper_implementations","repo_kind":"listed","path":"labml_nn/graphs/gatv2/experiment.py","file_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/HEAD/labml_nn/graphs/gatv2/experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"365d918744e8f7d5"}},{"code_sha256_prefix":"d6b15de3f7df93c6","entry":"GraphAttentionV2Layer","repo":"labmlai/annotated_deep_learning_paper_implementations","repo_kind":"listed","path":"labml_nn/graphs/gatv2/experiment.py","file_url":"https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/HEAD/labml_nn/graphs/gatv2/experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6b15de3f7df93c6"}},{"code_sha256_prefix":"7080188a5f04e65f","entry":"get_num_clusters","repo":"rusty1s/pytorch_geometric","repo_kind":"listed","path":"torch_geometric/datasets/hydro_net.py","file_url":"https://github.com/rusty1s/pytorch_geometric/blob/HEAD/torch_geometric/datasets/hydro_net.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7080188a5f04e65f"}},{"code_sha256_prefix":"0a76eaa4d06d2d91","entry":"map_annotation","repo":"rusty1s/pytorch_geometric","repo_kind":"listed","path":"torch_geometric/config_store.py","file_url":"https://github.com/rusty1s/pytorch_geometric/blob/HEAD/torch_geometric/config_store.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a76eaa4d06d2d91"}},{"code_sha256_prefix":"37b2364ea0f56557","entry":"read_energy","repo":"rusty1s/pytorch_geometric","repo_kind":"listed","path":"torch_geometric/datasets/hydro_net.py","file_url":"https://github.com/rusty1s/pytorch_geometric/blob/HEAD/torch_geometric/datasets/hydro_net.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"37b2364ea0f56557"}},{"code_sha256_prefix":"c19c99d92c84e9b8","entry":"use_segment_matmul_heuristic","repo":"rusty1s/pytorch_geometric","repo_kind":"listed","path":"torch_geometric/backend.py","file_url":"https://github.com/rusty1s/pytorch_geometric/blob/HEAD/torch_geometric/backend.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c19c99d92c84e9b8"}},{"code_sha256_prefix":"56cb5f68019f62a7","entry":"GATv2SelfAttention","repo":"machadoprx/GATv2-TensorFlow2","repo_kind":"listed","path":"gatv2_layer.py","file_url":"https://github.com/machadoprx/GATv2-TensorFlow2/blob/HEAD/gatv2_layer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56cb5f68019f62a7"}},{"code_sha256_prefix":"a56e8eaabb61634b","entry":"compile","repo":"rusty1s/pytorch_geometric","repo_kind":"listed","path":"torch_geometric/_compile.py","file_url":"https://github.com/rusty1s/pytorch_geometric/blob/HEAD/torch_geometric/_compile.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a56e8eaabb61634b"}},{"code_sha256_prefix":"49bec40bd43ba73d","entry":"get_auc_score","repo":"mahequn001/graphlearning","repo_kind":"listed","path":"model_zoo/gae/util.py","file_url":"https://github.com/mahequn001/graphlearning/blob/HEAD/model_zoo/gae/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"49bec40bd43ba73d"}},{"code_sha256_prefix":"1ede5ca46a7df870","entry":"pop_by_prefix","repo":"tensorflow/gnn","repo_kind":"listed","path":"tensorflow_gnn/graph/dict_utils.py","file_url":"https://github.com/tensorflow/gnn/blob/HEAD/tensorflow_gnn/graph/dict_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ede5ca46a7df870"}},{"code_sha256_prefix":"8349c934df099399","entry":"read_atoms","repo":"rusty1s/pytorch_geometric","repo_kind":"listed","path":"torch_geometric/datasets/hydro_net.py","file_url":"https://github.com/rusty1s/pytorch_geometric/blob/HEAD/torch_geometric/datasets/hydro_net.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8349c934df099399"}},{"code_sha256_prefix":"51ea4acdfd945732","entry":"safe_onnx_export","repo":"rusty1s/pytorch_geometric","repo_kind":"listed","path":"torch_geometric/_onnx.py","file_url":"https://github.com/rusty1s/pytorch_geometric/blob/HEAD/torch_geometric/_onnx.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51ea4acdfd945732"}},{"code_sha256_prefix":"977c76c93dace67a","entry":"to_dataclass","repo":"rusty1s/pytorch_geometric","repo_kind":"listed","path":"torch_geometric/config_store.py","file_url":"https://github.com/rusty1s/pytorch_geometric/blob/HEAD/torch_geometric/config_store.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"977c76c93dace67a"}},{"code_sha256_prefix":"1457b037fb82f82e","entry":"to_edge_list","repo":"tensorflow/gnn","repo_kind":"listed","path":"tensorflow_gnn/converters/networkx_io.py","file_url":"https://github.com/tensorflow/gnn/blob/HEAD/tensorflow_gnn/converters/networkx_io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1457b037fb82f82e"}},{"code_sha256_prefix":"abf6cd02b673f7f8","entry":"to_networkx_graph","repo":"tensorflow/gnn","repo_kind":"listed","path":"tensorflow_gnn/converters/networkx_io.py","file_url":"https://github.com/tensorflow/gnn/blob/HEAD/tensorflow_gnn/converters/networkx_io.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"abf6cd02b673f7f8"}},{"code_sha256_prefix":"36972132fe777084","entry":"with_key_prefix","repo":"tensorflow/gnn","repo_kind":"listed","path":"tensorflow_gnn/graph/dict_utils.py","file_url":"https://github.com/tensorflow/gnn/blob/HEAD/tensorflow_gnn/graph/dict_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"36972132fe777084"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}