{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/simmim-a-simple-framework-for-masked-image","title":"SimMIM: A Simple Framework for Masked Image Modeling","arxiv_id":"2111.09886","date":"2021-11-18","proceeding":"CVPR 2022 1","authors":["Zhenda Xie","Zheng Zhang","Yue Cao","Yutong Lin","Jianmin Bao","Zhuliang Yao","Qi Dai","Han Hu"],"abstract":"This paper presents SimMIM, a simple framework for masked image modeling. We simplify recently proposed related approaches without special designs such as block-wise masking and tokenization via discrete VAE or clustering. To study what let the masked image modeling task learn good representations, we systematically study the major components in our framework, and find that simple designs of each component have revealed very strong representation learning performance: 1) random masking of the input image with a moderately large masked patch size (e.g., 32) makes a strong pre-text task; 2) predicting raw pixels of RGB values by direct regression performs no worse than the patch classification approaches with complex designs; 3) the prediction head can be as light as a linear layer, with no worse performance than heavier ones. Using ViT-B, our approach achieves 83.8% top-1 fine-tuning accuracy on ImageNet-1K by pre-training also on this dataset, surpassing previous best approach by +0.6%. When applied on a larger model of about 650 million parameters, SwinV2-H, it achieves 87.1% top-1 accuracy on ImageNet-1K using only ImageNet-1K data. We also leverage this approach to facilitate the training of a 3B model (SwinV2-G), that by $40\\times$ less data than that in previous practice, we achieve the state-of-the-art on four representative vision benchmarks. The code and models will be publicly available at https://github.com/microsoft/SimMIM.","url_abs":"https://arxiv.org/abs/2111.09886v2","url_pdf":"https://arxiv.org/pdf/2111.09886v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"simmim-a-simple-framework-for-masked-image","repo_url":"https://github.com/microsoft/simmim","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"simmim-a-simple-framework-for-masked-image","repo_url":"https://github.com/Westlake-AI/openmixup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"simmim-a-simple-framework-for-masked-image","repo_url":"https://github.com/impiga/plain-detr","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"simmim-a-simple-framework-for-masked-image","repo_url":"https://github.com/isaaccorley/hydro-foundation-model","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"simmim-a-simple-framework-for-masked-image","repo_url":"https://github.com/lightly-ai/lightly","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"simmim-a-simple-framework-for-masked-image","repo_url":"https://github.com/Hazqeel09/ellzaf_ml","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"simmim-a-simple-framework-for-masked-image","repo_url":"https://gitlab.com/birder/birder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"task-2","task_name":"Task 2"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"SimMIM (SwinV2-H, 512)","rank_in_archive_order":10,"of":65,"metrics":{"Number of Params":"658M","Top 1 Accuracy":"87.1%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"SimMIM (Swin-L)","rank_in_archive_order":24,"of":65,"metrics":{"Number of Params":"197M","Top 1 Accuracy":"85.4%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"SimMIM (Swin-B)","rank_in_archive_order":38,"of":65,"metrics":{"Number of Params":"88M","Top 1 Accuracy":"84.0%"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"SimMIM (ViT-B/16)","rank_in_archive_order":43,"of":65,"metrics":{"Number of Params":"85M","Top 1 Accuracy":"83.8%"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.09886","atlas_url":"https://app.syntology.ai/?focus=2111.09886","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.09886"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Westlake-AI/openmixup","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/microsoft/SimMIM","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lightly-ai/lightly","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Hazqeel09/ellzaf_ml","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://gitlab.com/birder/birder","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/impiga/plain-detr","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/isaaccorley/hydro-foundation-model","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/simmim","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":8,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":13,"ran":8,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b0e55e16e4ab4b04","entry":"GlobalAttention","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0e55e16e4ab4b04"}},{"code_sha256_prefix":"6d0ff69e341e91b6","entry":"LayerNormFP32","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d0ff69e341e91b6"}},{"code_sha256_prefix":"ef982aba178be7dd","entry":"LinearFP32","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef982aba178be7dd"}},{"code_sha256_prefix":"804ddd3c6093f073","entry":"PatchEmbed","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"804ddd3c6093f073"}},{"code_sha256_prefix":"a5518beb9daf7d60","entry":"PatchMerging","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5518beb9daf7d60"}},{"code_sha256_prefix":"fddc32909150f889","entry":"SimMIM","repo":"microsoft/simmim","repo_kind":"official","path":"models/simmim.py","file_url":"https://github.com/microsoft/simmim/blob/HEAD/models/simmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fddc32909150f889"}},{"code_sha256_prefix":"9ce7427c4503c13e","entry":"SimMIM","repo":"Hazqeel09/ellzaf_ml","repo_kind":"listed","path":"ellzaf_ml/models/simmim.py","file_url":"https://github.com/Hazqeel09/ellzaf_ml/blob/HEAD/ellzaf_ml/models/simmim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9ce7427c4503c13e"}},{"code_sha256_prefix":"5eec152e243ad651","entry":"WindowAttention","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5eec152e243ad651"}},{"code_sha256_prefix":"4e2c8a04e709451b","entry":"custom_normalize","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4e2c8a04e709451b"}},{"code_sha256_prefix":"590e8cc54154effc","entry":"BasicLayer","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"590e8cc54154effc"}},{"code_sha256_prefix":"bcf8d3634776d004","entry":"SwinTransformerBlock","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bcf8d3634776d004"}},{"code_sha256_prefix":"deaa894233f1e3da","entry":"SwinTransformerGlobalBlock","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"deaa894233f1e3da"}},{"code_sha256_prefix":"d346a9e5a95c653f","entry":"SwinTransformerV2","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d346a9e5a95c653f"}},{"code_sha256_prefix":"6feca106254274b3","entry":"load_swinv2_checkpoint","repo":"impiga/plain-detr","repo_kind":"listed","path":"models/swin_transformer_v2.py","file_url":"https://github.com/impiga/plain-detr/blob/HEAD/models/swin_transformer_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6feca106254274b3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}