{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/energy-transformer","title":"Energy Transformer","arxiv_id":"2302.07253","date":"2023-02-14","proceeding":"NeurIPS 2023 11","authors":["Benjamin Hoover","Yuchen Liang","Bao Pham","Rameswar Panda","Hendrik Strobelt","Duen Horng Chau","Mohammed J. Zaki","Dmitry Krotov"],"abstract":"Our work combines aspects of three promising paradigms in machine learning, namely, attention mechanism, energy-based models, and associative memory. Attention is the power-house driving modern deep learning successes, but it lacks clear theoretical foundations. Energy-based models allow a principled approach to discriminative and generative tasks, but the design of the energy functional is not straightforward. At the same time, Dense Associative Memory models or Modern Hopfield Networks have a well-established theoretical foundation, and allow an intuitive design of the energy function. We propose a novel architecture, called the Energy Transformer (or ET for short), that uses a sequence of attention layers that are purposely designed to minimize a specifically engineered energy function, which is responsible for representing the relationships between the tokens. In this work, we introduce the theoretical foundations of ET, explore its empirical capabilities using the image completion task, and obtain strong quantitative results on the graph anomaly detection and graph classification tasks.","url_abs":"https://arxiv.org/abs/2302.07253v2","url_pdf":"https://arxiv.org/pdf/2302.07253v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"energy-transformer","repo_url":"https://github.com/bhoov/energy-transformer-jax","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"energy-transformer","repo_url":"https://github.com/zhuergou/energy-transformer-for-graph-anomaly-detection","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"energy-transformer","repo_url":"https://github.com/Lemon-cmd/energy-transformer-graph","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null},{"paper_slug":"energy-transformer","repo_url":"https://github.com/Lemon-cmd/energy-transformer-torch","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"anomaly-detection","task_name":"Anomaly Detection"},{"task_slug":"graph-anomaly-detection","task_name":"Graph Anomaly Detection"},{"task_slug":"graph-classification","task_name":"Graph Classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.07253","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.07253"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Lemon-cmd/energy-transformer-torch","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhuergou/energy-transformer-for-graph-anomaly-detection","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Lemon-cmd/energy-transformer-graph","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bhoov/energy-transformer-jax","reach":null}],"summary":{"ran_fixture":2,"ran_honours":2,"ran_draft_wrong":4,"unverified":5},"by_repo_kind":{"official":{"samples":11,"ran":6,"repositories":4}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"75dcdb1c3263beb4","entry":"array_to_img","repo":"bhoov/energy-transformer-jax","repo_kind":"official","path":"image_core.py","file_url":"https://github.com/bhoov/energy-transformer-jax/blob/HEAD/image_core.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"75dcdb1c3263beb4"}},{"code_sha256_prefix":"9f8fb6db436eff3e","entry":"check_type","repo":"Lemon-cmd/energy-transformer-graph","repo_kind":"official","path":"src/model/et.py","file_url":"https://github.com/Lemon-cmd/energy-transformer-graph/blob/HEAD/src/model/et.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f8fb6db436eff3e"}},{"code_sha256_prefix":"f6b944f50d3f15ae","entry":"count_parameters","repo":"Lemon-cmd/energy-transformer-torch","repo_kind":"official","path":"image_et/utils.py","file_url":"https://github.com/Lemon-cmd/energy-transformer-torch/blob/HEAD/image_et/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f6b944f50d3f15ae"}},{"code_sha256_prefix":"e232328a8a2b8b09","entry":"custom_message_func","repo":"zhuergou/energy-transformer-for-graph-anomaly-detection","repo_kind":"official","path":"energy_transformer_layer.py","file_url":"https://github.com/zhuergou/energy-transformer-for-graph-anomaly-detection/blob/HEAD/energy_transformer_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e232328a8a2b8b09"}},{"code_sha256_prefix":"43e7507240634b11","entry":"img_to_array","repo":"bhoov/energy-transformer-jax","repo_kind":"official","path":"image_core.py","file_url":"https://github.com/bhoov/energy-transformer-jax/blob/HEAD/image_core.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"43e7507240634b11"}},{"code_sha256_prefix":"d67b0a557c3733b2","entry":"normal","repo":"Lemon-cmd/energy-transformer-graph","repo_kind":"official","path":"src/model/et.py","file_url":"https://github.com/Lemon-cmd/energy-transformer-graph/blob/HEAD/src/model/et.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d67b0a557c3733b2"}},{"code_sha256_prefix":"654acacaf00954fb","entry":"scaled_exp","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"654acacaf00954fb"}},{"code_sha256_prefix":"23fba400cff6fafa","entry":"src_dot_dst","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"23fba400cff6fafa"}},{"code_sha256_prefix":"47b4e637f9d9e8f9","entry":"gen_mask_id","repo":"Lemon-cmd/energy-transformer-torch","repo_kind":"official","path":"image_et/utils.py","file_url":"https://github.com/Lemon-cmd/energy-transformer-torch/blob/HEAD/image_et/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47b4e637f9d9e8f9"}},{"code_sha256_prefix":"0c3b6f391746cf8b","entry":"get_files","repo":"Lemon-cmd/energy-transformer-torch","repo_kind":"official","path":"image_et/misc.py","file_url":"https://github.com/Lemon-cmd/energy-transformer-torch/blob/HEAD/image_et/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0c3b6f391746cf8b"}},{"code_sha256_prefix":"3a6bd7dc66242ce6","entry":"get_latest_file","repo":"Lemon-cmd/energy-transformer-torch","repo_kind":"official","path":"image_et/misc.py","file_url":"https://github.com/Lemon-cmd/energy-transformer-torch/blob/HEAD/image_et/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a6bd7dc66242ce6"}},{"code_sha256_prefix":"a389ee3bf2f9a9dc","entry":"parse_number","repo":"Lemon-cmd/energy-transformer-torch","repo_kind":"official","path":"image_et/misc.py","file_url":"https://github.com/Lemon-cmd/energy-transformer-torch/blob/HEAD/image_et/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a389ee3bf2f9a9dc"}},{"code_sha256_prefix":"d62e6f8bc4ddac46","entry":"unnormalize","repo":"Lemon-cmd/energy-transformer-torch","repo_kind":"official","path":"image_et/utils.py","file_url":"https://github.com/Lemon-cmd/energy-transformer-torch/blob/HEAD/image_et/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d62e6f8bc4ddac46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}