{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/recasting-self-attention-with-holographic","title":"Recasting Self-Attention with Holographic Reduced Representations","arxiv_id":"2305.19534","date":"2023-05-31","proceeding":null,"authors":["Mohammad Mahmudul Alam","Edward Raff","Stella Biderman","Tim Oates","James Holt"],"abstract":"In recent years, self-attention has become the dominant paradigm for sequence modeling in a variety of domains. However, in domains with very long sequence lengths the $\\mathcal{O}(T^2)$ memory and $\\mathcal{O}(T^2 H)$ compute costs can make using transformers infeasible. Motivated by problems in malware detection, where sequence lengths of $T \\geq 100,000$ are a roadblock to deep learning, we re-cast self-attention using the neuro-symbolic approach of Holographic Reduced Representations (HRR). In doing so we perform the same high-level strategy of the standard self-attention: a set of queries matching against a set of keys, and returning a weighted response of the values for each key. Implemented as a ``Hrrformer'' we obtain several benefits including $\\mathcal{O}(T H \\log H)$ time complexity, $\\mathcal{O}(T H)$ space complexity, and convergence in $10\\times$ fewer epochs. Nevertheless, the Hrrformer achieves near state-of-the-art accuracy on LRA benchmarks and we are able to learn with just a single layer. Combined, these benefits make our Hrrformer the first viable Transformer for such long malware classification sequences and up to $280\\times$ faster to train on the Long Range Arena benchmark. Code is available at \\url{https://github.com/NeuromorphicComputationResearchProgram/Hrrformer}","url_abs":"https://arxiv.org/abs/2305.19534v1","url_pdf":"https://arxiv.org/pdf/2305.19534v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"recasting-self-attention-with-holographic","repo_url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"malware-classification","task_name":"Malware Classification"},{"task_slug":"malware-detection","task_name":"Malware Detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2305.19534","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.19534"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":8,"ran":0,"repositories":1},"community":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"11b73c012ece44e7","entry":"cross_entropy_loss","repo":"neuromorphiccomputationresearchprogram/hrrformer","repo_kind":"official","path":"image/hrrformer_mgpu.py","file_url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer/blob/HEAD/image/hrrformer_mgpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11b73c012ece44e7"}},{"code_sha256_prefix":"f0b8db0a03177b1a","entry":"evaluate","repo":"neuromorphiccomputationresearchprogram/hrrformer","repo_kind":"official","path":"image/hrrformer_mgpu.py","file_url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer/blob/HEAD/image/hrrformer_mgpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0b8db0a03177b1a"}},{"code_sha256_prefix":"a98da1a55cc51929","entry":"initialize_model","repo":"FutureComputing4AI/Hrrformer","repo_kind":"community","path":"malware/hrrformer_mgpu.py","file_url":"https://github.com/FutureComputing4AI/Hrrformer/blob/HEAD/malware/hrrformer_mgpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a98da1a55cc51929"}},{"code_sha256_prefix":"4f1a5274115568d4","entry":"initialize_model","repo":"neuromorphiccomputationresearchprogram/hrrformer","repo_kind":"official","path":"image/hrrformer_mgpu.py","file_url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer/blob/HEAD/image/hrrformer_mgpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4f1a5274115568d4"}},{"code_sha256_prefix":"c574416e3c30908b","entry":"initialize_model","repo":"neuromorphiccomputationresearchprogram/hrrformer","repo_kind":"official","path":"retrieval/hrrformer_mgpu.py","file_url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer/blob/HEAD/retrieval/hrrformer_mgpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c574416e3c30908b"}},{"code_sha256_prefix":"387fdc05497309c2","entry":"look_ahead_mask","repo":"neuromorphiccomputationresearchprogram/hrrformer","repo_kind":"official","path":"utils.py","file_url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"387fdc05497309c2"}},{"code_sha256_prefix":"9332d0930ef1d545","entry":"merge","repo":"neuromorphiccomputationresearchprogram/hrrformer","repo_kind":"official","path":"utils.py","file_url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9332d0930ef1d545"}},{"code_sha256_prefix":"38dc6299f38b0c74","entry":"pad_collate_func","repo":"neuromorphiccomputationresearchprogram/hrrformer","repo_kind":"official","path":"malware/binaryLoader.py","file_url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer/blob/HEAD/malware/binaryLoader.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38dc6299f38b0c74"}},{"code_sha256_prefix":"a8de53c24dc4ea75","entry":"predict","repo":"FutureComputing4AI/Hrrformer","repo_kind":"community","path":"malware/hrrformer_mgpu.py","file_url":"https://github.com/FutureComputing4AI/Hrrformer/blob/HEAD/malware/hrrformer_mgpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a8de53c24dc4ea75"}},{"code_sha256_prefix":"aaa1f91613c8db28","entry":"split","repo":"neuromorphiccomputationresearchprogram/hrrformer","repo_kind":"official","path":"utils.py","file_url":"https://github.com/neuromorphiccomputationresearchprogram/hrrformer/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aaa1f91613c8db28"}},{"code_sha256_prefix":"73cf404c6a2e8e98","entry":"train_step","repo":"FutureComputing4AI/Hrrformer","repo_kind":"community","path":"malware/hrrformer_mgpu.py","file_url":"https://github.com/FutureComputing4AI/Hrrformer/blob/HEAD/malware/hrrformer_mgpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73cf404c6a2e8e98"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}