{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-self-supervised-vision-transformers","title":"Efficient Self-supervised Vision Transformers for Representation Learning","arxiv_id":"2106.09785","date":"2021-06-17","proceeding":"ICLR 2022 4","authors":["Chunyuan Li","Jianwei Yang","Pengchuan Zhang","Mei Gao","Bin Xiao","Xiyang Dai","Lu Yuan","Jianfeng Gao"],"abstract":"This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. Second, we propose a new pre-training task of region matching which allows the model to capture fine-grained region dependencies and as a result significantly improves the quality of the learned vision representations. Our results show that combining the two techniques, EsViT achieves 81.3% top-1 on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput. When transferring to downstream linear classification tasks, EsViT outperforms its supervised counterpart on 17 out of 18 datasets. The code and models are publicly available: https://github.com/microsoft/esvit","url_abs":"https://arxiv.org/abs/2106.09785v2","url_pdf":"https://arxiv.org/pdf/2106.09785v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-self-supervised-vision-transformers","repo_url":"https://github.com/microsoft/esvit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"}],"methods":[{"method_slug":"esvit","method_name":"EsViT"}],"datasets_introduced":[],"methods_introduced":[{"slug":"esvit","name":"EsViT","full_name":"EsViT"}],"results":[{"leaderboard":"/sota/self-supervised-image-classification-on","task":"Self-Supervised Image Classification","dataset":"ImageNet","model":"EsViT (Swin-B)","rank_in_archive_order":15,"of":144,"metrics":{"Number of Params":"87M","Top 1 Accuracy":"81.3","Top 5 Accuracy":"95.5"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on","task":"Self-Supervised Image Classification","dataset":"ImageNet","model":"EsViT(Swin-S)","rank_in_archive_order":19,"of":144,"metrics":{"Number of Params":"49M","Top 1 Accuracy":"80.8"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"EsViT (Swin-B)","rank_in_archive_order":40,"of":65,"metrics":{"Number of Params":"87M","Top 1 Accuracy":"83.9%"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.09785","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.09785"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/esvit","reach":null}],"summary":{"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f7fcf540ff7b9a1e","entry":"PatchEmbed","repo":"microsoft/esvit","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/microsoft/esvit/blob/HEAD/models/swin_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7fcf540ff7b9a1e"}},{"code_sha256_prefix":"fd8f760f4a030218","entry":"PatchMerging","repo":"microsoft/esvit","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/microsoft/esvit/blob/HEAD/models/swin_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fd8f760f4a030218"}},{"code_sha256_prefix":"c3e0b48845da21b7","entry":"BasicLayer","repo":"microsoft/esvit","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/microsoft/esvit/blob/HEAD/models/swin_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3e0b48845da21b7"}},{"code_sha256_prefix":"09f9c77057fb0a2e","entry":"SwinTransformer","repo":"microsoft/esvit","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/microsoft/esvit/blob/HEAD/models/swin_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09f9c77057fb0a2e"}},{"code_sha256_prefix":"65e2245eb3e718e0","entry":"SwinTransformerBlock","repo":"microsoft/esvit","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/microsoft/esvit/blob/HEAD/models/swin_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65e2245eb3e718e0"}},{"code_sha256_prefix":"9951a2e090a99f11","entry":"WindowAttention","repo":"microsoft/esvit","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/microsoft/esvit/blob/HEAD/models/swin_transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9951a2e090a99f11"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}