{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/demystify-mamba-in-vision-a-linear-attention","title":"Demystify Mamba in Vision: A Linear Attention Perspective","arxiv_id":"2405.16605","date":"2024-05-26","proceeding":null,"authors":["Dongchen Han","Ziyi Wang","Zhuofan Xia","Yizeng Han","Yifan Pu","Chunjiang Ge","Jun Song","Shiji Song","Bo Zheng","Gao Huang"],"abstract":"Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba's success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba's success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-Inspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https://github.com/LeapLabTHU/MLLA.","url_abs":"https://arxiv.org/abs/2405.16605v2","url_pdf":"https://arxiv.org/pdf/2405.16605v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"demystify-mamba-in-vision-a-linear-attention","repo_url":"https://github.com/LeapLabTHU/MLLA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2405.16605","atlas_url":"https://app.syntology.ai/?focus=2405.16605","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16605"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/LeapLabTHU/MLLA","reach":null}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e67f3c92243ab692","entry":"ConvLayer","repo":"LeapLabTHU/MLLA","repo_kind":"official","path":"models/mlla.py","file_url":"https://github.com/LeapLabTHU/MLLA/blob/HEAD/models/mlla.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e67f3c92243ab692"}},{"code_sha256_prefix":"145a5aeec8a513b8","entry":"LinearAttention","repo":"LeapLabTHU/MLLA","repo_kind":"official","path":"models/mlla.py","file_url":"https://github.com/LeapLabTHU/MLLA/blob/HEAD/models/mlla.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"145a5aeec8a513b8"}},{"code_sha256_prefix":"81956cbe9ee28d73","entry":"MLLABlock","repo":"LeapLabTHU/MLLA","repo_kind":"official","path":"models/mlla.py","file_url":"https://github.com/LeapLabTHU/MLLA/blob/HEAD/models/mlla.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81956cbe9ee28d73"}},{"code_sha256_prefix":"aea9c35eb5503710","entry":"PatchMerging","repo":"LeapLabTHU/MLLA","repo_kind":"official","path":"models/mlla.py","file_url":"https://github.com/LeapLabTHU/MLLA/blob/HEAD/models/mlla.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aea9c35eb5503710"}},{"code_sha256_prefix":"688729cd9ab4f05e","entry":"RoPE","repo":"LeapLabTHU/MLLA","repo_kind":"official","path":"models/mlla.py","file_url":"https://github.com/LeapLabTHU/MLLA/blob/HEAD/models/mlla.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"688729cd9ab4f05e"}},{"code_sha256_prefix":"158e1c2f6afa4d41","entry":"Stem","repo":"LeapLabTHU/MLLA","repo_kind":"official","path":"models/mlla.py","file_url":"https://github.com/LeapLabTHU/MLLA/blob/HEAD/models/mlla.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"158e1c2f6afa4d41"}},{"code_sha256_prefix":"8a4668090d89154b","entry":"BasicLayer","repo":"LeapLabTHU/MLLA","repo_kind":"official","path":"models/mlla.py","file_url":"https://github.com/LeapLabTHU/MLLA/blob/HEAD/models/mlla.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a4668090d89154b"}},{"code_sha256_prefix":"83ecb75e85115f04","entry":"MLLA","repo":"LeapLabTHU/MLLA","repo_kind":"official","path":"models/mlla.py","file_url":"https://github.com/LeapLabTHU/MLLA/blob/HEAD/models/mlla.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83ecb75e85115f04"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}