{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/outlier-efficient-hopfield-layers-for-large","title":"Outlier-Efficient Hopfield Layers for Large Transformer-Based Models","arxiv_id":"2404.03828","date":"2024-04-04","proceeding":null,"authors":["Jerry Yao-Chieh Hu","Pei-Hsuan Chang","Robin Luo","Hong-Yu Chen","Weijian Li","Wei-Po Wang","Han Liu"],"abstract":"We introduce an Outlier-Efficient Modern Hopfield Model (termed $\\mathrm{OutEffHop}$) and use it to address the outlier inefficiency problem of {training} gigantic transformer-based models. Our main contribution is a novel associative memory model facilitating \\textit{outlier-efficient} associative memory retrievals. Interestingly, this memory model manifests a model-based interpretation of an outlier-efficient attention mechanism (${\\rm Softmax}_1$): it is an approximation of the memory retrieval process of $\\mathrm{OutEffHop}$. Methodologically, this allows us to introduce novel outlier-efficient Hopfield layers as powerful alternatives to traditional attention mechanisms, with superior post-quantization performance. Theoretically, the Outlier-Efficient Modern Hopfield Model retains and improves the desirable properties of standard modern Hopfield models, including fixed point convergence and exponential storage capacity. Empirically, we demonstrate the efficacy of the proposed model across large-scale transformer-based and Hopfield-based models (including BERT, OPT, ViT, and STanHop-Net), benchmarking against state-of-the-art methods like $\\mathtt{Clipped\\_Softmax}$ and $\\mathtt{Gated\\_Attention}$. Notably, $\\mathrm{OutEffHop}$ achieves an average reduction of 22+\\% in average kurtosis and 26+\\% in the maximum infinity norm of model outputs across four models. Code is available at \\href{https://github.com/MAGICS-LAB/OutEffHop}{GitHub}; models are on \\href{https://huggingface.co/collections/magicslabnu/outeffhop-6610fcede8d2cda23009a98f}{Hugging Face Hub}; future updates are on \\href{https://arxiv.org/abs/2404.03828}{arXiv}.","url_abs":"https://arxiv.org/abs/2404.03828v2","url_pdf":"https://arxiv.org/pdf/2404.03828v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"outlier-efficient-hopfield-layers-for-large","repo_url":"https://github.com/magics-lab/outeffhop","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"quantization","task_name":"Quantization"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"opt","method_name":"OPT"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/benchmarking-on-wiki-40b","task":"Benchmarking","dataset":"Wiki-40B","model":"OutEffHop-Bert_base","rank_in_archive_order":1,"of":1,"metrics":{"Perplexity":"6.209"},"uses_additional_data":false},{"leaderboard":"/sota/quantization-on-wiki-40b","task":"Quantization","dataset":"Wiki-40B","model":"OutEffHop-Bert_base","rank_in_archive_order":1,"of":1,"metrics":{"Perplexity":"6.295"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.03828","atlas_url":"https://app.syntology.ai/?focus=2404.03828","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.03828"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/magics-lab/outeffhop","reach":null}],"summary":{"ran":5,"ran_draft_wrong":5,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e7b18cd260d16db1","entry":"AlphaChooser","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7b18cd260d16db1"}},{"code_sha256_prefix":"4f7463a0aff83e2a","entry":"ClipSoftmax","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f7463a0aff83e2a"}},{"code_sha256_prefix":"9d1fce5d64cdaaf3","entry":"EntmaxAlpha","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d1fce5d64cdaaf3"}},{"code_sha256_prefix":"d6305b9e14fc507d","entry":"EntmaxBisectFunction","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d6305b9e14fc507d"}},{"code_sha256_prefix":"aba43948313580df","entry":"Softmax_1","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aba43948313580df"}},{"code_sha256_prefix":"1e2438ad229df84b","entry":"clipped_softmax","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e2438ad229df84b"}},{"code_sha256_prefix":"e9e2c444e97f90ba","entry":"clipped_softmax_1","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9e2c444e97f90ba"}},{"code_sha256_prefix":"2d3be2464ee960b7","entry":"entmax_bisect","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d3be2464ee960b7"}},{"code_sha256_prefix":"580329f21ac41e9e","entry":"softmax_1","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"580329f21ac41e9e"}},{"code_sha256_prefix":"9bbad4b677349ef5","entry":"softmax_n_shifted_zeros","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9bbad4b677349ef5"}},{"code_sha256_prefix":"1b3b7daf7cbbd041","entry":"Association","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b3b7daf7cbbd041"}},{"code_sha256_prefix":"cc3dee79e09eb57e","entry":"ClipSoftmax_1","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cc3dee79e09eb57e"}},{"code_sha256_prefix":"0e0d879508ab2c99","entry":"Hopfield","repo":"magics-lab/outeffhop","repo_kind":"official","path":"STanHop_time_seeries/cross_models/hopfield.py","file_url":"https://github.com/magics-lab/outeffhop/blob/HEAD/STanHop_time_seeries/cross_models/hopfield.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e0d879508ab2c99"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}