{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/equiformer-equivariant-graph-attention","title":"Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs","arxiv_id":"2206.11990","date":"2022-06-23","proceeding":null,"authors":["Yi-Lun Liao","Tess Smidt"],"abstract":"Despite their widespread success in various domains, Transformer networks have yet to perform well across datasets in the domain of 3D atomistic graphs such as molecules even when 3D-related inductive biases like translational invariance and rotational equivariance are considered. In this paper, we demonstrate that Transformers can generalize well to 3D atomistic graphs and present Equiformer, a graph neural network leveraging the strength of Transformer architectures and incorporating SE(3)/E(3)-equivariant features based on irreducible representations (irreps). First, we propose a simple and effective architecture by only replacing original operations in Transformers with their equivariant counterparts and including tensor products. Using equivariant operations enables encoding equivariant information in channels of irreps features without complicating graph structures. With minimal modifications to Transformers, this architecture has already achieved strong empirical results. Second, we propose a novel attention mechanism called equivariant graph attention, which improves upon typical attention in Transformers through replacing dot product attention with multi-layer perceptron attention and including non-linear message passing. With these two innovations, Equiformer achieves competitive results to previous models on QM9, MD17 and OC20 datasets.","url_abs":"https://arxiv.org/abs/2206.11990v2","url_pdf":"https://arxiv.org/pdf/2206.11990v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"equiformer-equivariant-graph-attention","repo_url":"https://github.com/atomicarchitects/equiformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"equiformer-equivariant-graph-attention","repo_url":"https://github.com/atomicarchitects/dens","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"equiformer-equivariant-graph-attention","repo_url":"https://github.com/atomicarchitects/equiformer_v2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"equiformer-equivariant-graph-attention","repo_url":"https://github.com/lucidrains/equiformer-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"graph-attention","task_name":"Graph Attention"},{"task_slug":"graph-neural-network","task_name":"Graph Neural Network"},{"task_slug":"graph-property-prediction","task_name":"Graph Property Prediction"},{"task_slug":"initial-structure-to-relaxed-energy-is2re-1","task_name":"Initial Structure to Relaxed Energy (IS2RE), Direct"},{"task_slug":"molecular-property-prediction","task_name":"Molecular Property Prediction"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"graph-neural-network","method_name":"Graph Neural Network"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/graph-property-prediction-on-qm9","task":"Graph Property Prediction","dataset":"QM9","model":"Equiformer","rank_in_archive_order":5,"of":10,"metrics":{"Standardized MAE":"0.70","alpha (ma)":"46","gap (meV)":"30","logMAE":"-5.82"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2206.11990","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2206.11990"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/atomicarchitects/equiformer_v2","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/atomicarchitects/equiformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/atomicarchitects/dens","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/equiformer-pytorch","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fd4284f5532a264d","entry":"get_mul_0","repo":"atomicarchitects/equiformer","repo_kind":"official","path":"nets/graph_attention_transformer.py","file_url":"https://github.com/atomicarchitects/equiformer/blob/HEAD/nets/graph_attention_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd4284f5532a264d"}},{"code_sha256_prefix":"122e87ee0b05894f","entry":"pack_one","repo":"lucidrains/equiformer-pytorch","repo_kind":"listed","path":"equiformer_pytorch/equiformer_pytorch.py","file_url":"https://github.com/lucidrains/equiformer-pytorch/blob/HEAD/equiformer_pytorch/equiformer_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"122e87ee0b05894f"}},{"code_sha256_prefix":"b8149022c0d28e97","entry":"unpack_one","repo":"lucidrains/equiformer-pytorch","repo_kind":"listed","path":"equiformer_pytorch/equiformer_pytorch.py","file_url":"https://github.com/lucidrains/equiformer-pytorch/blob/HEAD/equiformer_pytorch/equiformer_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b8149022c0d28e97"}},{"code_sha256_prefix":"22ae6ca5b382e2d0","entry":"get_norm_layer","repo":"atomicarchitects/equiformer","repo_kind":"official","path":"nets/graph_attention_transformer.py","file_url":"https://github.com/atomicarchitects/equiformer/blob/HEAD/nets/graph_attention_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22ae6ca5b382e2d0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}