{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-linear-attention-meets-autoregressive","title":"When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models","arxiv_id":"2406.07368","date":"2024-06-11","proceeding":null,"authors":["Haoran You","Yichao Fu","Zheng Wang","Amir Yazdanbakhsh","Yingyan Celine Lin"],"abstract":"Autoregressive Large Language Models (LLMs) have achieved impressive performance in language tasks but face two significant bottlenecks: (1) quadratic complexity in the attention module as the number of tokens increases, and (2) limited efficiency due to the sequential processing nature of autoregressive LLMs during generation. While linear attention and speculative decoding offer potential solutions, their applicability and synergistic potential for enhancing autoregressive LLMs remain uncertain. We conduct the first comprehensive study on the efficacy of existing linear attention methods for autoregressive LLMs, integrating them with speculative decoding. We introduce an augmentation technique for linear attention that ensures compatibility with speculative decoding, enabling more efficient training and serving of LLMs. Extensive experiments and ablation studies involving seven existing linear attention models and five encoder/decoder-based LLMs consistently validate the effectiveness of our augmented linearized LLMs. Notably, our approach achieves up to a 6.67 reduction in perplexity on the LLaMA model and up to a 2$\\times$ speedup during generation compared to prior linear attention methods. Codes and models are available at https://github.com/GATECH-EIC/Linearized-LLM.","url_abs":"https://arxiv.org/abs/2406.07368v2","url_pdf":"https://arxiv.org/pdf/2406.07368v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-linear-attention-meets-autoregressive","repo_url":"https://github.com/gatech-eic/linearized-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"}],"methods":[{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.07368","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.07368"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gatech-eic/linearized-llm","reach":null}],"summary":{"ran_honours":1,"ran_draft_wrong":3,"ran_fixture":1,"ran_violates":2,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"5f4fdf97a20b1819","entry":"padding_to_multiple_of","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"flash_pytorch.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/flash_pytorch.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5f4fdf97a20b1819"}},{"code_sha256_prefix":"5a9b970426ac2f96","entry":"_global_linear_attn_causal","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"experiments/llama_attn_replace.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/experiments/llama_attn_replace.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5a9b970426ac2f96"}},{"code_sha256_prefix":"288bee86546fbf67","entry":"_global_linear_attn_p","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"experiments/llama_attn_replace.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/experiments/llama_attn_replace.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"288bee86546fbf67"}},{"code_sha256_prefix":"60fff7c3c400d7ff","entry":"default","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"60fff7c3c400d7ff"}},{"code_sha256_prefix":"2107ce5c9a769398","entry":"eval_decorator","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"autoregressive_wrapper.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/autoregressive_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2107ce5c9a769398"}},{"code_sha256_prefix":"b767065fd213438a","entry":"exists","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"autoregressive_wrapper.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/autoregressive_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b767065fd213438a"}},{"code_sha256_prefix":"5d77a7df8b8103e4","entry":"top_k","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"autoregressive_wrapper.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/autoregressive_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d77a7df8b8103e4"}},{"code_sha256_prefix":"bef480ba7a34dcdd","entry":"_local_quadratic_attn_p","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"experiments/llama_attn_replace.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/experiments/llama_attn_replace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bef480ba7a34dcdd"}},{"code_sha256_prefix":"d65e3eb5da171762","entry":"_local_quadratic_attn_p_g1","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"experiments/llama_attn_replace.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/experiments/llama_attn_replace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d65e3eb5da171762"}},{"code_sha256_prefix":"682226da161991da","entry":"linear_forward_noflashattn","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"experiments/llama_attn_replace.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/experiments/llama_attn_replace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"682226da161991da"}},{"code_sha256_prefix":"6e35e1c26b0f21f6","entry":"yoso_pointwise","repo":"gatech-eic/linearized-llm","repo_kind":"official","path":"experiments/llama_attn_replace.py","file_url":"https://github.com/gatech-eic/linearized-llm/blob/HEAD/experiments/llama_attn_replace.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e35e1c26b0f21f6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}