{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gated-slot-attention-for-efficient-linear","title":"Gated Slot Attention for Efficient Linear-Time Sequence Modeling","arxiv_id":"2409.07146","date":"2024-09-11","proceeding":null,"authors":["Yu Zhang","Songlin Yang","Ruijie Zhu","Yue Zhang","Leyang Cui","Yiqiao Wang","Bolun Wang","Freda Shi","Bailin Wang","Wei Bi","Peng Zhou","Guohong Fu"],"abstract":"Linear attention Transformers and their gated variants, celebrated for enabling parallel training and efficient recurrent inference, still fall short in recall-intensive tasks compared to traditional Transformers and demand significant resources for training from scratch. This paper introduces Gated Slot Attention (GSA), which enhances Attention with Bounded-memory-Control (ABC) by incorporating a gating mechanism inspired by Gated Linear Attention (GLA). Essentially, GSA comprises a two-layer GLA linked via $\\operatorname{softmax}$, utilizing context-aware memory reading and adaptive forgetting to improve memory capacity while maintaining compact recurrent state size. This design greatly enhances both training and inference efficiency through GLA's hardware-efficient training algorithm and reduced state size. Additionally, retaining the $\\operatorname{softmax}$ operation is particularly beneficial in \"finetuning pretrained Transformers to RNNs\" (T2R) settings, reducing the need for extensive training from scratch. Extensive experiments confirm GSA's superior performance in scenarios requiring in-context recall and in T2R settings.","url_abs":"https://arxiv.org/abs/2409.07146v2","url_pdf":"https://arxiv.org/pdf/2409.07146v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gated-slot-attention-for-efficient-linear","repo_url":"https://github.com/sustcsonglin/flash-linear-attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"gated-slot-attention-for-efficient-linear","repo_url":"https://github.com/fla-org/flash-linear-attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"gated-slot-attention-for-efficient-linear","repo_url":"https://github.com/torchrwkv/flash-linear-attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2409.07146","atlas_url":"https://app.syntology.ai/?focus=2409.07146","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}