{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-shift-operation-meets-vision-transformer","title":"When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism","arxiv_id":"2201.10801","date":"2022-01-26","proceeding":null,"authors":["Guangting Wang","Yucheng Zhao","Chuanxin Tang","Chong Luo","Wenjun Zeng"],"abstract":"Attention mechanism has been widely believed as the key to success of vision transformers (ViTs), since it provides a flexible and powerful way to model spatial relationships. However, is the attention mechanism truly an indispensable part of ViT? Can it be replaced by some other alternatives? To demystify the role of attention mechanism, we simplify it into an extremely simple case: ZERO FLOP and ZERO parameter. Concretely, we revisit the shift operation. It does not contain any parameter or arithmetic calculation. The only operation is to exchange a small portion of the channels between neighboring features. Based on this simple operation, we construct a new backbone network, namely ShiftViT, where the attention layers in ViT are substituted by shift operations. Surprisingly, ShiftViT works quite well in several mainstream tasks, e.g., classification, detection, and segmentation. The performance is on par with or even better than the strong baseline Swin Transformer. These results suggest that the attention mechanism might not be the vital factor that makes ViT successful. It can be even replaced by a zero-parameter operation. We should pay more attentions to the remaining parts of ViT in the future work. Code is available at github.com/microsoft/SPACH.","url_abs":"https://arxiv.org/abs/2201.10801v1","url_pdf":"https://arxiv.org/pdf/2201.10801v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-shift-operation-meets-vision-transformer","repo_url":"https://github.com/microsoft/SPACH","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"when-shift-operation-meets-vision-transformer","repo_url":"https://github.com/keras-team/keras-io/blob/master/examples/vision/shiftvit.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"stochastic-depth","method_name":"Stochastic Depth"},{"method_slug":"swin-transformer","method_name":"Swin Transformer"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"Shift-B","rank_in_archive_order":446,"of":1060,"metrics":{"GFLOPs":"15.2","Number of params":"88M","Top 1 Accuracy":"83.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"Shift-S","rank_in_archive_order":502,"of":1060,"metrics":{"GFLOPs":"8.5","Number of params":"50M","Top 1 Accuracy":"82.8%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"Shift-T","rank_in_archive_order":618,"of":1060,"metrics":{"GFLOPs":"4.4","Number of params":"28M","Top 1 Accuracy":"81.7%"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco-minival","task":"Object Detection","dataset":"COCO minival","model":"Shift-T","rank_in_archive_order":219,"of":220,"metrics":{"APM":"42.3"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Shift-B (UperNet)","rank_in_archive_order":137,"of":235,"metrics":{"Validation mIoU":"49.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Shift-B","rank_in_archive_order":156,"of":235,"metrics":{"Validation mIoU":"47.9"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Shift-S","rank_in_archive_order":157,"of":235,"metrics":{"Validation mIoU":"47.8"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Shift-T","rank_in_archive_order":177,"of":235,"metrics":{"Validation mIoU":"46.3"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2201.10801","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2201.10801"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/keras-team/keras-io/blob/master/examples/vision/shiftvit.py","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/SPACH","reach":null}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"025a8da133d73582","entry":"BasicLayer","repo":"microsoft/SPACH","repo_kind":"official","path":"models/shiftvit.py","file_url":"https://github.com/microsoft/SPACH/blob/HEAD/models/shiftvit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"025a8da133d73582"}},{"code_sha256_prefix":"468dbc110847fd17","entry":"GroupNorm","repo":"microsoft/SPACH","repo_kind":"official","path":"models/shiftvit.py","file_url":"https://github.com/microsoft/SPACH/blob/HEAD/models/shiftvit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"468dbc110847fd17"}},{"code_sha256_prefix":"eb2bb570ac51f183","entry":"Mlp","repo":"microsoft/SPACH","repo_kind":"official","path":"models/shiftvit.py","file_url":"https://github.com/microsoft/SPACH/blob/HEAD/models/shiftvit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eb2bb570ac51f183"}},{"code_sha256_prefix":"d079baf8601e41e5","entry":"PatchEmbed","repo":"microsoft/SPACH","repo_kind":"official","path":"models/shiftvit.py","file_url":"https://github.com/microsoft/SPACH/blob/HEAD/models/shiftvit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d079baf8601e41e5"}},{"code_sha256_prefix":"9d0d874136a0ef10","entry":"PatchMerging","repo":"microsoft/SPACH","repo_kind":"official","path":"models/shiftvit.py","file_url":"https://github.com/microsoft/SPACH/blob/HEAD/models/shiftvit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d0d874136a0ef10"}},{"code_sha256_prefix":"c1151f8745d4efba","entry":"ShiftViTBlock","repo":"microsoft/SPACH","repo_kind":"official","path":"models/shiftvit.py","file_url":"https://github.com/microsoft/SPACH/blob/HEAD/models/shiftvit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1151f8745d4efba"}},{"code_sha256_prefix":"296bef5bd4e63d1b","entry":"ShiftViT","repo":"microsoft/SPACH","repo_kind":"official","path":"models/shiftvit.py","file_url":"https://github.com/microsoft/SPACH/blob/HEAD/models/shiftvit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"296bef5bd4e63d1b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}