{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/geometric-transformer-with-interatomic","title":"Geometric Transformer with Interatomic Positional Encoding","arxiv_id":null,"date":"2023-09-21","proceeding":"NeurIPS 2023 11","authors":[],"abstract":"The widespread adoption of Transformer architectures in various data modalities has opened new avenues for the applications in molecular modeling. \nNevertheless, it remains elusive that whether the Transformer-based architecture can do molecular modeling as good as equivariant GNNs. \nIn this paper, by designing Interatomic Positional Encoding (IPE) that\nparameterizes atomic environments as Transformer's positional encodings,\nwe propose Geoformer, a novel geometric Transformer \nto effectively model molecular structures for various molecular property prediction. \nWe evaluate Geoformer on several benchmarks, including the QM9 dataset and the recently proposed Molecule3D dataset. \nCompared with both Transformers and equivariant GNN models, Geoformer outperforms the state-of-the-art (SoTA) algorithms on QM9, and achieves the best performance on Molecule3D for both random and scaffold splits.\nBy introducing IPE, \nGeoformer paves the way for molecular geometric modeling based on Transformer architecture.","url_abs":"https://openreview.net/forum?id=9o6KQrklrE","url_pdf":"https://openreview.net/pdf?id=9o6KQrklrE","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"geometric-transformer-with-interatomic","repo_url":"https://github.com/microsoft/AI2BMD","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}