{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/linear-attention-for-efficient-bidirectional","title":"Linear Attention for Efficient Bidirectional Sequence Modeling","arxiv_id":"2502.16249","date":"2025-02-22","proceeding":null,"authors":["Arshia Afzal","Elias Abad Rocamora","Leyla Naz Candogan","Pol Puigdemont","Francesco Tonin","Yongtao Wu","Mahsa Shoaran","Volkan Cevher"],"abstract":"Transformers with linear attention enable fast and parallel training. Moreover, they can be formulated as Recurrent Neural Networks (RNNs), for efficient linear-time inference. While extensively evaluated in causal sequence modeling, they have yet to be extended to the bidirectional setting. This work introduces the LION framework, establishing new theoretical foundations for linear transformers in bidirectional sequence modeling. LION constructs a bidirectional RNN equivalent to full Linear Attention. This extends the benefits of linear transformers: parallel training, and efficient inference, into the bidirectional setting. Using LION, we cast three linear transformers to their bidirectional form: LION-LIT, the bidirectional variant corresponding to (Katharopoulos et al., 2020); LION-D, extending RetNet (Sun et al., 2023); and LION-S, a linear transformer with a stable selective mask inspired by selectivity of SSMs (Dao & Gu, 2024). Replacing the attention block with LION (-LIT, -D, -S) achieves performance on bidirectional tasks that approaches that of Transformers and State-Space Models (SSMs), while delivering significant improvements in training speed. Our implementation is available in http://github.com/LIONS-EPFL/LION.","url_abs":"https://arxiv.org/abs/2502.16249v1","url_pdf":"https://arxiv.org/pdf/2502.16249v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"linear-attention-for-efficient-bidirectional","repo_url":"https://github.com/lions-epfl/lion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"lion","method_name":"Lion"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.16249","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.16249"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/HazyResearch/m2","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lions-epfl/lion","reach":null}],"summary":{"ran_draft_wrong":5,"ran_fixture":5,"ran":1,"ran_honours":3},"by_repo_kind":{"official":{"samples":14,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"173c2a794682a779","entry":"Casual_Mask_Decay_Partial","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"173c2a794682a779"}},{"code_sha256_prefix":"74d6e1e65a29b46c","entry":"Casual_Mask_Selective_Partial","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"74d6e1e65a29b46c"}},{"code_sha256_prefix":"b7d6d1185456f649","entry":"Causal_Mask_Decay","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b7d6d1185456f649"}},{"code_sha256_prefix":"ff5e874a04ef822f","entry":"Causal_Mask_Selective","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ff5e874a04ef822f"}},{"code_sha256_prefix":"0fd20524afba58d9","entry":"Lion_Attention","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0fd20524afba58d9"}},{"code_sha256_prefix":"96ac937123b116ca","entry":"compute_curve_order","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"96ac937123b116ca"}},{"code_sha256_prefix":"4ef8a90a9ec9a0f6","entry":"coords_to_index","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4ef8a90a9ec9a0f6"}},{"code_sha256_prefix":"233f32ffc1bfedb1","entry":"create_matrix_from_tensor","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"233f32ffc1bfedb1"}},{"code_sha256_prefix":"2c6f64b164f56782","entry":"create_matrix_from_tensor_backward","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2c6f64b164f56782"}},{"code_sha256_prefix":"663e12d457f7ff82","entry":"create_matrix_from_tensor_forward","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"663e12d457f7ff82"}},{"code_sha256_prefix":"90399293d39e23ac","entry":"elu_shifted","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"90399293d39e23ac"}},{"code_sha256_prefix":"882f6b4910fe627d","entry":"index_to_coords_indexes","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"882f6b4910fe627d"}},{"code_sha256_prefix":"5745d11adac3446b","entry":"s_curve","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5745d11adac3446b"}},{"code_sha256_prefix":"19c5151cad5ba8a1","entry":"silu_shifted","repo":"lions-epfl/lion","repo_kind":"official","path":"img_classification/lion/models_lion.py","file_url":"https://github.com/lions-epfl/lion/blob/HEAD/img_classification/lion/models_lion.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"19c5151cad5ba8a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}