{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/slab-efficient-transformers-with-simplified","title":"SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization","arxiv_id":"2405.11582","date":"2024-05-19","proceeding":null,"authors":["Jialong Guo","Xinghao Chen","Yehui Tang","Yunhe Wang"],"abstract":"Transformers have become foundational architectures for both natural language and computer vision tasks. However, the high computational cost makes it quite challenging to deploy on resource-constraint devices. This paper investigates the computational bottleneck modules of efficient transformer, i.e., normalization layers and attention modules. LayerNorm is commonly used in transformer architectures but is not computational friendly due to statistic calculation during inference. However, replacing LayerNorm with more efficient BatchNorm in transformer often leads to inferior performance and collapse in training. To address this problem, we propose a novel method named PRepBN to progressively replace LayerNorm with re-parameterized BatchNorm in training. Moreover, we propose a simplified linear attention (SLA) module that is simple yet effective to achieve strong performance. Extensive experiments on image classification as well as object detection demonstrate the effectiveness of our proposed method. For example, our SLAB-Swin obtains $83.6\\%$ top-1 accuracy on ImageNet-1K with $16.2$ms latency, which is $2.4$ms less than that of Flatten-Swin with $0.1\\%$ higher accuracy. We also evaluated our method for language modeling task and obtain comparable performance and lower latency.Codes are publicly available at https://github.com/xinghaochen/SLAB and https://github.com/mindspore-lab/models/tree/master/research/huawei-noah/SLAB.","url_abs":"https://arxiv.org/abs/2405.11582v2","url_pdf":"https://arxiv.org/pdf/2405.11582v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"slab-efficient-transformers-with-simplified","repo_url":"https://github.com/xinghaochen/slab","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"slab-efficient-transformers-with-simplified","repo_url":"https://github.com/mindspore-lab/models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"paper_slug":"slab-efficient-transformers-with-simplified","repo_url":"https://github.com/MS-P3/code4/tree/main/SLAB","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.11582","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.11582"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xinghaochen/slab","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-lab/models","reach":{"status":"gone","observed_at":"2026-09-17","how":"tree_404+repo_404"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MS-P3/code4/tree/main/SLAB","reach":null}],"summary":{"ran_fixture":2,"ran_draft_wrong":3,"ran":1,"unverified":9},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1},"listed":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"144d10b49baeb8a6","entry":"window_partition","repo":"xinghaochen/slab","repo_kind":"official","path":"classification/models/slab_swin.py","file_url":"https://github.com/xinghaochen/slab/blob/HEAD/classification/models/slab_swin.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"144d10b49baeb8a6"}},{"code_sha256_prefix":"39b18f062e0cc6d9","entry":"img2windows","repo":"xinghaochen/slab","repo_kind":"official","path":"classification/models/slab_cswin.py","file_url":"https://github.com/xinghaochen/slab/blob/HEAD/classification/models/slab_cswin.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39b18f062e0cc6d9"}},{"code_sha256_prefix":"b9e24c3e3e0e88a7","entry":"merge","repo":"MS-P3/code4","repo_kind":"listed","path":"DECO/model_utils/config.py","file_url":"https://github.com/MS-P3/code4/blob/HEAD/DECO/model_utils/config.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b9e24c3e3e0e88a7"}},{"code_sha256_prefix":"1720c87bb3c19836","entry":"transpose","repo":"MS-P3/code4","repo_kind":"listed","path":"CIGF/DataHandler.py","file_url":"https://github.com/MS-P3/code4/blob/HEAD/CIGF/DataHandler.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1720c87bb3c19836"}},{"code_sha256_prefix":"61bf152e6a42a184","entry":"window_reverse","repo":"xinghaochen/slab","repo_kind":"official","path":"classification/models/slab_swin.py","file_url":"https://github.com/xinghaochen/slab/blob/HEAD/classification/models/slab_swin.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61bf152e6a42a184"}},{"code_sha256_prefix":"ab01fbfe58c75b08","entry":"windows2img","repo":"xinghaochen/slab","repo_kind":"official","path":"classification/models/slab_cswin.py","file_url":"https://github.com/xinghaochen/slab/blob/HEAD/classification/models/slab_cswin.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab01fbfe58c75b08"}},{"code_sha256_prefix":"07e8b4b487c0abb6","entry":"build_scheduler","repo":"xinghaochen/slab","repo_kind":"official","path":"classification/lr_scheduler.py","file_url":"https://github.com/xinghaochen/slab/blob/HEAD/classification/lr_scheduler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07e8b4b487c0abb6"}},{"code_sha256_prefix":"82e52feafbf040bb","entry":"create_logger","repo":"xinghaochen/slab","repo_kind":"official","path":"classification/logger.py","file_url":"https://github.com/xinghaochen/slab/blob/HEAD/classification/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"82e52feafbf040bb"}},{"code_sha256_prefix":"aa92af09147be134","entry":"getParam","repo":"MS-P3/code4","repo_kind":"listed","path":"CIGF/Utils/NNLayers.py","file_url":"https://github.com/MS-P3/code4/blob/HEAD/CIGF/Utils/NNLayers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aa92af09147be134"}},{"code_sha256_prefix":"20681a9065c0a731","entry":"get_config","repo":"xinghaochen/slab","repo_kind":"official","path":"classification/config.py","file_url":"https://github.com/xinghaochen/slab/blob/HEAD/classification/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20681a9065c0a731"}},{"code_sha256_prefix":"84f996d9ed4e6952","entry":"negSamp","repo":"MS-P3/code4","repo_kind":"listed","path":"CIGF/DataHandler.py","file_url":"https://github.com/MS-P3/code4/blob/HEAD/CIGF/DataHandler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"84f996d9ed4e6952"}},{"code_sha256_prefix":"85b09f6f153fd5d3","entry":"parse_cli_to_yaml","repo":"MS-P3/code4","repo_kind":"listed","path":"DECO/model_utils/config.py","file_url":"https://github.com/MS-P3/code4/blob/HEAD/DECO/model_utils/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"85b09f6f153fd5d3"}},{"code_sha256_prefix":"96a9939953d5992d","entry":"parse_yaml","repo":"MS-P3/code4","repo_kind":"listed","path":"DECO/model_utils/config.py","file_url":"https://github.com/MS-P3/code4/blob/HEAD/DECO/model_utils/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"96a9939953d5992d"}},{"code_sha256_prefix":"fceaa2d7c28f6761","entry":"str2bool","repo":"MS-P3/code4","repo_kind":"listed","path":"D2Co/prepare_data.py","file_url":"https://github.com/MS-P3/code4/blob/HEAD/D2Co/prepare_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fceaa2d7c28f6761"}},{"code_sha256_prefix":"68c1e936b1101784","entry":"transToLsts","repo":"MS-P3/code4","repo_kind":"listed","path":"CIGF/DataHandler.py","file_url":"https://github.com/MS-P3/code4/blob/HEAD/CIGF/DataHandler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68c1e936b1101784"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}