{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/equivariant-neural-functional-networks-for","title":"Equivariant Neural Functional Networks for Transformers","arxiv_id":"2410.04209","date":"2024-10-05","proceeding":null,"authors":["Viet-Hoang Tran","Thieu N. Vo","An Nguyen The","Tho Tran Huu","Minh-Khoi Nguyen-Nhat","Thanh Tran","Duy-Tung Pham","Tan Minh Nguyen"],"abstract":"This paper systematically explores neural functional networks (NFN) for transformer architectures. NFN are specialized neural networks that treat the weights, gradients, or sparsity patterns of a deep neural network (DNN) as input data and have proven valuable for tasks such as learnable optimizers, implicit data representations, and weight editing. While NFN have been extensively developed for MLP and CNN, no prior work has addressed their design for transformers, despite the importance of transformers in modern deep learning. This paper aims to address this gap by providing a systematic study of NFN for transformers. We first determine the maximal symmetric group of the weights in a multi-head attention module as well as a necessary and sufficient condition under which two sets of hyperparameters of the multi-head attention module define the same function. We then define the weight space of transformer architectures and its associated group action, which leads to the design principles for NFN in transformers. Based on these, we introduce Transformer-NFN, an NFN that is equivariant under this group action. Additionally, we release a dataset of more than 125,000 Transformers model checkpoints trained on two datasets with two different tasks, providing a benchmark for evaluating Transformer-NFN and encouraging further research on transformer training and performance.","url_abs":"https://arxiv.org/abs/2410.04209v1","url_pdf":"https://arxiv.org/pdf/2410.04209v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"nfn","method_name":"NFN"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.04209","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.04209"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/MathematicalAI-NUS/Transformer-NFN","reach":null}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c72bb4157d7c9e3a","entry":"ArraySpec","repo":"MathematicalAI-NUS/Transformer-NFN","repo_kind":"found_in_text","path":"nfn_transformer/layers/layers.py","file_url":"https://github.com/MathematicalAI-NUS/Transformer-NFN/blob/HEAD/nfn_transformer/layers/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c72bb4157d7c9e3a"}},{"code_sha256_prefix":"3ec27b376a35582b","entry":"AttentionNetworkSpec","repo":"MathematicalAI-NUS/Transformer-NFN","repo_kind":"found_in_text","path":"nfn_transformer/layers/layers.py","file_url":"https://github.com/MathematicalAI-NUS/Transformer-NFN/blob/HEAD/nfn_transformer/layers/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ec27b376a35582b"}},{"code_sha256_prefix":"5828069cdb018584","entry":"AttentionWeightSpaceFeatures","repo":"MathematicalAI-NUS/Transformer-NFN","repo_kind":"found_in_text","path":"nfn_transformer/layers/layers.py","file_url":"https://github.com/MathematicalAI-NUS/Transformer-NFN/blob/HEAD/nfn_transformer/layers/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5828069cdb018584"}},{"code_sha256_prefix":"c8057e5c908926cf","entry":"EinsumLayer","repo":"MathematicalAI-NUS/Transformer-NFN","repo_kind":"found_in_text","path":"nfn_transformer/layers/layers.py","file_url":"https://github.com/MathematicalAI-NUS/Transformer-NFN/blob/HEAD/nfn_transformer/layers/layers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c8057e5c908926cf"}},{"code_sha256_prefix":"463c8ed5b198900a","entry":"TransformersLinear","repo":"MathematicalAI-NUS/Transformer-NFN","repo_kind":"found_in_text","path":"nfn_transformer/layers/layers.py","file_url":"https://github.com/MathematicalAI-NUS/Transformer-NFN/blob/HEAD/nfn_transformer/layers/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"463c8ed5b198900a"}},{"code_sha256_prefix":"4c1d262382f31c1f","entry":"set_init_einsum_","repo":"MathematicalAI-NUS/Transformer-NFN","repo_kind":"found_in_text","path":"nfn_transformer/layers/layers.py","file_url":"https://github.com/MathematicalAI-NUS/Transformer-NFN/blob/HEAD/nfn_transformer/layers/layers.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4c1d262382f31c1f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}