{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cswin-transformer-a-general-vision","title":"CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows","arxiv_id":"2107.00652","date":"2021-07-01","proceeding":"CVPR 2022 1","authors":["Xiaoyi Dong","Jianmin Bao","Dongdong Chen","Weiming Zhang","Nenghai Yu","Lu Yuan","Dong Chen","Baining Guo"],"abstract":"We present CSWin Transformer, an efficient and effective Transformer-based backbone for general-purpose vision tasks. A challenging issue in Transformer design is that global self-attention is very expensive to compute whereas local self-attention often limits the field of interactions of each token. To address this issue, we develop the Cross-Shaped Window self-attention mechanism for computing self-attention in the horizontal and vertical stripes in parallel that form a cross-shaped window, with each stripe obtained by splitting the input feature into stripes of equal width. We provide a mathematical analysis of the effect of the stripe width and vary the stripe width for different layers of the Transformer network which achieves strong modeling capability while limiting the computation cost. We also introduce Locally-enhanced Positional Encoding (LePE), which handles the local positional information better than existing encoding schemes. LePE naturally supports arbitrary input resolutions, and is thus especially effective and friendly for downstream tasks. Incorporated with these designs and a hierarchical structure, CSWin Transformer demonstrates competitive performance on common vision tasks. Specifically, it achieves 85.4\\% Top-1 accuracy on ImageNet-1K without any extra training data or label, 53.9 box AP and 46.4 mask AP on the COCO detection task, and 52.2 mIOU on the ADE20K semantic segmentation task, surpassing previous state-of-the-art Swin Transformer backbone by +1.2, +2.0, +1.4, and +2.0 respectively under the similar FLOPs setting. By further pretraining on the larger dataset ImageNet-21K, we achieve 87.5% Top-1 accuracy on ImageNet-1K and high segmentation performance on ADE20K with 55.7 mIoU. The code and models are available at https://github.com/microsoft/CSWin-Transformer.","url_abs":"https://arxiv.org/abs/2107.00652v3","url_pdf":"https://arxiv.org/pdf/2107.00652v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cswin-transformer-a-general-vision","repo_url":"https://github.com/microsoft/CSWin-Transformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"cswin-transformer-a-general-vision","repo_url":"https://github.com/TJUdyk/CSWin-Transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"cswin-transformer-a-general-vision","repo_url":"https://github.com/fogfog2/packnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"cswin-transformer-a-general-vision","repo_url":"https://github.com/lyqcom/cswin-transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"cswin-transformer-a-general-vision","repo_url":"https://github.com/BR-IDL/PaddleViT/blob/main/image_classification/CSwin","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null},{"paper_slug":"cswin-transformer-a-general-vision","repo_url":"https://github.com/PaddlePaddle/PaddleClas","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"cswin-transformer-a-general-vision","repo_url":"https://gitlab.com/birder/birder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"stochastic-depth","method_name":"Stochastic Depth"},{"method_slug":"swin-transformer","method_name":"Swin Transformer"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CSWin-L (384 res,ImageNet-22k pretrain)","rank_in_archive_order":84,"of":1060,"metrics":{"GFLOPs":"96.8","Number of params":"173M","Top 1 Accuracy":"87.5%"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"CSWin-L (UperNet, ImageNet-22k pretrain)","rank_in_archive_order":45,"of":235,"metrics":{"Validation mIoU":"55.70"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k-val","task":"Semantic Segmentation","dataset":"ADE20K val","model":"CSWin-L (UperNet, ImageNet-22k pretrain)","rank_in_archive_order":27,"of":95,"metrics":{"mIoU":"55.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2107.00652","atlas_url":"https://app.syntology.ai/?focus=2107.00652","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.00652"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lyqcom/cswin-transformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/PaddlePaddle/PaddleClas","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BR-IDL/PaddleViT/blob/main/image_classification/CSwin","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/TJUdyk/CSWin-Transformer","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fogfog2/packnet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/CSWin-Transformer","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://gitlab.com/birder/birder","reach":null}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"39b18f062e0cc6d9","entry":"img2windows","repo":"lyqcom/cswin-transformer","repo_kind":"listed","path":"models/cswin.py","file_url":"https://github.com/lyqcom/cswin-transformer/blob/HEAD/models/cswin.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39b18f062e0cc6d9"}},{"code_sha256_prefix":"ab01fbfe58c75b08","entry":"windows2img","repo":"lyqcom/cswin-transformer","repo_kind":"listed","path":"models/cswin.py","file_url":"https://github.com/lyqcom/cswin-transformer/blob/HEAD/models/cswin.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab01fbfe58c75b08"}},{"code_sha256_prefix":"3b22aba5effad739","entry":"create_optimizer","repo":"microsoft/CSWin-Transformer","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/microsoft/CSWin-Transformer/blob/HEAD/finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b22aba5effad739"}},{"code_sha256_prefix":"4d0c2350b24a4430","entry":"load_img","repo":"microsoft/CSWin-Transformer","repo_kind":"official","path":"labeled_memcached_dataset.py","file_url":"https://github.com/microsoft/CSWin-Transformer/blob/HEAD/labeled_memcached_dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d0c2350b24a4430"}},{"code_sha256_prefix":"e3b8df6691b9c559","entry":"train_epoch","repo":"microsoft/CSWin-Transformer","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/microsoft/CSWin-Transformer/blob/HEAD/finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3b8df6691b9c559"}},{"code_sha256_prefix":"e2a98c6a53ded11f","entry":"validate","repo":"microsoft/CSWin-Transformer","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/microsoft/CSWin-Transformer/blob/HEAD/finetune.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e2a98c6a53ded11f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}