{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/masked-attention-is-all-you-need-for-graphs","title":"An end-to-end attention-based approach for learning on graphs","arxiv_id":"2402.10793","date":"2024-02-16","proceeding":null,"authors":["David Buterez","Jon Paul Janet","Dino Oglic","Pietro Lio"],"abstract":"There has been a recent surge in transformer-based architectures for learning on graphs, mainly motivated by attention as an effective learning mechanism and the desire to supersede handcrafted operators characteristic of message passing schemes. However, concerns over their empirical effectiveness, scalability, and complexity of the pre-processing steps have been raised, especially in relation to much simpler graph neural networks that typically perform on par with them across a wide range of benchmarks. To tackle these shortcomings, we consider graphs as sets of edges and propose a purely attention-based approach consisting of an encoder and an attention pooling mechanism. The encoder vertically interleaves masked and vanilla self-attention modules to learn an effective representations of edges, while allowing for tackling possible misspecifications in input graphs. Despite its simplicity, the approach outperforms fine-tuned message passing baselines and recently proposed transformer-based methods on more than 70 node and graph-level tasks, including challenging long-range benchmarks. Moreover, we demonstrate state-of-the-art performance across different tasks, ranging from molecular to vision graphs, and heterophilous node classification. The approach also outperforms graph neural networks and transformers in transfer learning settings, and scales much better than alternatives with a similar performance level or expressive power.","url_abs":"https://arxiv.org/abs/2402.10793v2","url_pdf":"https://arxiv.org/pdf/2402.10793v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"masked-attention-is-all-you-need-for-graphs","repo_url":"https://github.com/davidbuterez/edge-set-attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"graph-classification","task_name":"Graph Classification"},{"task_slug":"graph-regression","task_name":"Graph Regression"},{"task_slug":"molecular-property-prediction","task_name":"Molecular Property Prediction"},{"task_slug":"node-classification","task_name":"Node Classification"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-pooling","method_name":"Attention Pooling"},{"method_slug":"graph-self-attention","method_name":"Graph Self-Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/graph-classification-on-cifar10-100k","task":"Graph Classification","dataset":"CIFAR10 100k","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":6,"of":20,"metrics":{"Accuracy (%)":"75.413±0.248"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-dd","task":"Graph Classification","dataset":"D&D","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":3,"of":53,"metrics":{"Accuracy":"83.529±1.743"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-enzymes","task":"Graph Classification","dataset":"ENZYMES","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":1,"of":54,"metrics":{"Accuracy":"79.423±1.658"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-imdb-b","task":"Graph Classification","dataset":"IMDb-B","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":2,"of":51,"metrics":{"Accuracy":"86.250±0.957"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-mnist","task":"Graph Classification","dataset":"MNIST","model":"ESA (Edge set attention, no positional encodings, tuned)","rank_in_archive_order":1,"of":13,"metrics":{"Accuracy":"98.917±0.020"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-mnist","task":"Graph Classification","dataset":"MNIST","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":3,"of":13,"metrics":{"Accuracy":"98.753±0.041"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-malnet-tiny","task":"Graph Classification","dataset":"MalNet-Tiny","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":1,"of":4,"metrics":{"Accuracy":"94.800±0.424","MCC":"0.935±0.005"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-nci1","task":"Graph Classification","dataset":"NCI1","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":2,"of":69,"metrics":{"Accuracy":"87.835±0.644"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-nci109","task":"Graph Classification","dataset":"NCI109","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":3,"of":38,"metrics":{"Accuracy":"84.976±0.551"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-proteins","task":"Graph Classification","dataset":"PROTEINS","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":4,"of":103,"metrics":{"Accuracy":"82.679±0.799"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-peptides-func","task":"Graph Classification","dataset":"Peptides-func","model":"ESA + RWSE (Edge set attention, Random Walk Structural Encoding, + validation set)","rank_in_archive_order":1,"of":44,"metrics":{"AP":"0.7479"},"uses_additional_data":true},{"leaderboard":"/sota/graph-classification-on-peptides-func","task":"Graph Classification","dataset":"Peptides-func","model":"ESA + RWSE (Edge set attention, Random Walk Structural Encoding, tuned)","rank_in_archive_order":3,"of":44,"metrics":{"AP":"0.7357±0.0036"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-peptides-func","task":"Graph Classification","dataset":"Peptides-func","model":"ESA (Edge set attention, no positional encodings, tuned)","rank_in_archive_order":13,"of":44,"metrics":{"AP":"0.7071±0.0015"},"uses_additional_data":false},{"leaderboard":"/sota/graph-classification-on-peptides-func","task":"Graph Classification","dataset":"Peptides-func","model":"ESA (Edge set attention, no positional encodings, not tuned)","rank_in_archive_order":19,"of":44,"metrics":{"AP":"0.6863±0.0044"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-esr2","task":"Graph Regression","dataset":"ESR2","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":1,"of":9,"metrics":{"R2":"0.697±0.000","RMSE":"0.486±0.697"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-f2","task":"Graph Regression","dataset":"F2","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":1,"of":9,"metrics":{"R2":"0.891±0.000","RMSE":"0.335±0.891"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-kit","task":"Graph Regression","dataset":"KIT","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":2,"of":9,"metrics":{"R2":"0.841±0.000","RMSE":"0.433±0.841"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-lipophilicity","task":"Graph Regression","dataset":"Lipophilicity","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":5,"of":23,"metrics":{"R2":"0.809±0.008","RMSE":"0.552±0.012"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-parp1","task":"Graph Regression","dataset":"PARP1","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":1,"of":9,"metrics":{"R2":"0.925±0.000","RMSE":"0.343±0.925"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-pcqm4mv2-lsc","task":"Graph Regression","dataset":"PCQM4Mv2-LSC","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":1,"of":20,"metrics":{"Test MAE":"N/A","Validation MAE":"0.0235"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-pgr","task":"Graph Regression","dataset":"PGR","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":1,"of":9,"metrics":{"R2":"0.725±0.000","RMSE":"0.507±0.725"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-peptides-struct","task":"Graph Regression","dataset":"Peptides-struct","model":"ESA + RWSE (Edge set attention, Random Walk Structural Encoding, tuned)","rank_in_archive_order":1,"of":39,"metrics":{"MAE":"0.2393±0.0004"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-peptides-struct","task":"Graph Regression","dataset":"Peptides-struct","model":"ESA (Edge set attention, no positional encodings, not tuned)","rank_in_archive_order":8,"of":39,"metrics":{"MAE":"0.2453±0.0003"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-zinc","task":"Graph Regression","dataset":"ZINC","model":"ESA + rings + NodeRWSE + EdgeRWSE","rank_in_archive_order":1,"of":27,"metrics":{"MAE":"0.051"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-zinc-500k","task":"Graph Regression","dataset":"ZINC-500k","model":"ESA + rings + NodeRWSE + EdgeRWSE","rank_in_archive_order":1,"of":36,"metrics":{"MAE":"0.051"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-zinc-full","task":"Graph Regression","dataset":"ZINC-full","model":"ESA + rings + NodeRWSE + EdgeRWSE","rank_in_archive_order":1,"of":19,"metrics":{"Test MAE":"0.0109±0.0002"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-zinc-full","task":"Graph Regression","dataset":"ZINC-full","model":"ESA + RWSE + CY2C (Edge set attention, Random Walk Structural Encoding, clique adjacency, tuned)","rank_in_archive_order":2,"of":19,"metrics":{"Test MAE":"0.0122±0.0004"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-zinc-full","task":"Graph Regression","dataset":"ZINC-full","model":"ESA + RWSE (Edge set attention, Random Walk Structural Encoding, tuned)","rank_in_archive_order":4,"of":19,"metrics":{"Test MAE":"0.0154±0.0001"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-zinc-full","task":"Graph Regression","dataset":"ZINC-full","model":"ESA + RWSE (Edge set attention, Random Walk Structural Encoding)","rank_in_archive_order":5,"of":19,"metrics":{"Test MAE":"0.017±0.001"},"uses_additional_data":false},{"leaderboard":"/sota/graph-regression-on-zinc-full","task":"Graph Regression","dataset":"ZINC-full","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":9,"of":19,"metrics":{"Test MAE":"0.027±0.001"},"uses_additional_data":false},{"leaderboard":"/sota/molecular-property-prediction-on-esol","task":"Molecular Property Prediction","dataset":"ESOL","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":1,"of":20,"metrics":{"R2":"0.944±0.002","RMSE":"0.485±0.009"},"uses_additional_data":false},{"leaderboard":"/sota/molecular-property-prediction-on-freesolv","task":"Molecular Property Prediction","dataset":"FreeSolv","model":"ESA (Edge set attention, no positional encodings)","rank_in_archive_order":1,"of":22,"metrics":{"R2":"0.977±0.001","RMSE":"0.595±0.013"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}