{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/luna-linear-unified-nested-attention","title":"Luna: Linear Unified Nested Attention","arxiv_id":"2106.01540","date":"2021-06-03","proceeding":"NeurIPS 2021 12","authors":["Xuezhe Ma","Xiang Kong","Sinong Wang","Chunting Zhou","Jonathan May","Hao Ma","Luke Zettlemoyer"],"abstract":"The quadratic computational and memory complexities of the Transformer's attention mechanism have limited its scalability for modeling long sequences. In this paper, we propose Luna, a linear unified nested attention mechanism that approximates softmax attention with two nested linear attention functions, yielding only linear (as opposed to quadratic) time and space complexity. Specifically, with the first attention function, Luna packs the input sequence into a sequence of fixed length. Then, the packed sequence is unpacked using the second attention function. As compared to a more traditional attention mechanism, Luna introduces an additional sequence with a fixed length as input and an additional corresponding output, which allows Luna to perform attention operation linearly, while also storing adequate contextual information. We perform extensive evaluations on three benchmarks of sequence modeling tasks: long-context sequence modeling, neural machine translation and masked language modeling for large-scale pretraining. Competitive or even better experimental results demonstrate both the effectiveness and efficiency of Luna compared to a variety","url_abs":"https://arxiv.org/abs/2106.01540v2","url_pdf":"https://arxiv.org/pdf/2106.01540v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"luna-linear-unified-nested-attention","repo_url":"https://github.com/XuezheMax/fairseq-apollo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"luna-linear-unified-nested-attention","repo_url":"https://github.com/sooftware/luna-transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"masked-language-modeling","task_name":"Masked Language Modeling"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.01540","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.01540"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/XuezheMax/fairseq-apollo","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sooftware/luna-transformer","reach":null}],"summary":{"ran":5,"unverified":5},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1},"listed":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"9b0ec916edfd0481","entry":"MultiHeadAttention","repo":"sooftware/luna-transformer","repo_kind":"listed","path":"luna_transformer/attention.py","file_url":"https://github.com/sooftware/luna-transformer/blob/HEAD/luna_transformer/attention.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b0ec916edfd0481"}},{"code_sha256_prefix":"07969f7041022050","entry":"distributed_init","repo":"XuezheMax/fairseq-apollo","repo_kind":"official","path":"fairseq/distributed_utils.py","file_url":"https://github.com/XuezheMax/fairseq-apollo/blob/HEAD/fairseq/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"07969f7041022050"}},{"code_sha256_prefix":"90b79fe15491de83","entry":"is_master","repo":"XuezheMax/fairseq-apollo","repo_kind":"official","path":"fairseq/distributed_utils.py","file_url":"https://github.com/XuezheMax/fairseq-apollo/blob/HEAD/fairseq/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"90b79fe15491de83"}},{"code_sha256_prefix":"82bdf940665aeeb4","entry":"safe_readline","repo":"XuezheMax/fairseq-apollo","repo_kind":"official","path":"fairseq/binarizer.py","file_url":"https://github.com/XuezheMax/fairseq-apollo/blob/HEAD/fairseq/binarizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"82bdf940665aeeb4"}},{"code_sha256_prefix":"9418b43048e4e4ec","entry":"url_to_filename","repo":"XuezheMax/fairseq-apollo","repo_kind":"official","path":"fairseq/file_utils.py","file_url":"https://github.com/XuezheMax/fairseq-apollo/blob/HEAD/fairseq/file_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"9418b43048e4e4ec"}},{"code_sha256_prefix":"f42cf74a0708c772","entry":"DotProductAttention","repo":"sooftware/luna-transformer","repo_kind":"listed","path":"luna_transformer/attention.py","file_url":"https://github.com/sooftware/luna-transformer/blob/HEAD/luna_transformer/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f42cf74a0708c772"}},{"code_sha256_prefix":"e29a0daaf387dc67","entry":"LinearUnifiedNestedAttention","repo":"sooftware/luna-transformer","repo_kind":"listed","path":"luna_transformer/attention.py","file_url":"https://github.com/sooftware/luna-transformer/blob/HEAD/luna_transformer/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e29a0daaf387dc67"}},{"code_sha256_prefix":"199e96db5edd0edd","entry":"all_reduce","repo":"XuezheMax/fairseq-apollo","repo_kind":"official","path":"fairseq/distributed_utils.py","file_url":"https://github.com/XuezheMax/fairseq-apollo/blob/HEAD/fairseq/distributed_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"199e96db5edd0edd"}},{"code_sha256_prefix":"b1a033619f614e17","entry":"filename_to_url","repo":"XuezheMax/fairseq-apollo","repo_kind":"official","path":"fairseq/file_utils.py","file_url":"https://github.com/XuezheMax/fairseq-apollo/blob/HEAD/fairseq/file_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1a033619f614e17"}},{"code_sha256_prefix":"bd3fe80662b28bc7","entry":"load_archive_file","repo":"XuezheMax/fairseq-apollo","repo_kind":"official","path":"fairseq/file_utils.py","file_url":"https://github.com/XuezheMax/fairseq-apollo/blob/HEAD/fairseq/file_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd3fe80662b28bc7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}