{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revealing-the-dark-secrets-of-masked-image","title":"Revealing the Dark Secrets of Masked Image Modeling","arxiv_id":"2205.13543","date":"2022-05-26","proceeding":"CVPR 2023 1","authors":["Zhenda Xie","Zigang Geng","Jingcheng Hu","Zheng Zhang","Han Hu","Yue Cao"],"abstract":"Masked image modeling (MIM) as pre-training is shown to be effective for numerous vision downstream tasks, but how and where MIM works remain unclear. In this paper, we compare MIM with the long-dominant supervised pre-trained models from two perspectives, the visualizations and the experiments, to uncover their key representational differences. From the visualizations, we find that MIM brings locality inductive bias to all layers of the trained models, but supervised models tend to focus locally at lower layers but more globally at higher layers. That may be the reason why MIM helps Vision Transformers that have a very large receptive field to optimize. Using MIM, the model can maintain a large diversity on attention heads in all layers. But for supervised models, the diversity on attention heads almost disappears from the last three layers and less diversity harms the fine-tuning performance. From the experiments, we find that MIM models can perform significantly better on geometric and motion tasks with weak semantics or fine-grained classification tasks, than their supervised counterparts. Without bells and whistles, a standard MIM pre-trained SwinV2-L could achieve state-of-the-art performance on pose estimation (78.9 AP on COCO test-dev and 78.0 AP on CrowdPose), depth estimation (0.287 RMSE on NYUv2 and 1.966 RMSE on KITTI), and video object tracking (70.7 SUC on LaSOT). For the semantic understanding datasets where the categories are sufficiently covered by the supervised pre-training, MIM models can still achieve highly competitive transfer performance. With a deeper understanding of MIM, we hope that our work can inspire new and solid research in this direction.","url_abs":"https://arxiv.org/abs/2205.13543v2","url_pdf":"https://arxiv.org/pdf/2205.13543v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revealing-the-dark-secrets-of-masked-image","repo_url":"https://github.com/SwinTransformer/MIM-Depth-Estimation","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"depth-estimation","task_name":"Depth Estimation"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"monocular-depth-estimation","task_name":"Monocular Depth Estimation"},{"task_slug":"object-tracking","task_name":"Object Tracking"},{"task_slug":"pose-estimation","task_name":"Pose Estimation"},{"task_slug":"video-object-tracking","task_name":"Video Object Tracking"},{"task_slug":"visual-object-tracking","task_name":"Visual Object Tracking"}],"methods":[{"method_slug":"mim","method_name":"MIM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/depth-estimation-on-nyu-depth-v2","task":"Depth Estimation","dataset":"NYU-Depth V2","model":"SwinV2-L 1K-MIM","rank_in_archive_order":3,"of":17,"metrics":{"RMS":"0.287"},"uses_additional_data":false},{"leaderboard":"/sota/depth-estimation-on-nyu-depth-v2","task":"Depth Estimation","dataset":"NYU-Depth V2","model":"SwinV2-B 1K-MIM","rank_in_archive_order":5,"of":17,"metrics":{"RMS":"0.304"},"uses_additional_data":false},{"leaderboard":"/sota/monocular-depth-estimation-on-kitti-eigen","task":"Monocular Depth Estimation","dataset":"KITTI Eigen split","model":"SwinV2-L 1K-MIM","rank_in_archive_order":17,"of":79,"metrics":{"Delta < 1.25":"0.977","Delta < 1.25^2":"0.998","Delta < 1.25^3":"1.000","RMSE":"1.966","RMSE log":"0.075","Sq Rel":"0.139","absolute relative error":"0.050"},"uses_additional_data":false},{"leaderboard":"/sota/monocular-depth-estimation-on-kitti-eigen","task":"Monocular Depth Estimation","dataset":"KITTI Eigen split","model":"SwinV2-B 1K-MIM","rank_in_archive_order":25,"of":79,"metrics":{"Delta < 1.25":"0.976","Delta < 1.25^2":"0.998","Delta < 1.25^3":"0.999","RMSE":"2.050","RMSE log":"0.078","Sq Rel":"0.148","absolute relative error":"0.052"},"uses_additional_data":false},{"leaderboard":"/sota/monocular-depth-estimation-on-nyu-depth-v2","task":"Monocular Depth Estimation","dataset":"NYU-Depth V2","model":"SwinV2-L 1K-MIM","rank_in_archive_order":29,"of":85,"metrics":{"Delta < 1.25":"0.949","Delta < 1.25^2":"0.994","Delta < 1.25^3":"0.999","RMSE":"0.287","absolute relative error":"0.083","log 10":"0.035"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-coco-test-dev","task":"Pose Estimation","dataset":"COCO test-dev","model":"SwinV2-L 1K-MIM","rank_in_archive_order":11,"of":47,"metrics":{"AP":"77.2"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-coco-test-dev","task":"Pose Estimation","dataset":"COCO test-dev","model":"SwinV2-B 1K-MIM","rank_in_archive_order":14,"of":47,"metrics":{"AP":"76.7"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-crowdpose","task":"Pose Estimation","dataset":"CrowdPose","model":"SwinV2-L 1K-MIM","rank_in_archive_order":4,"of":12,"metrics":{"AP":"75.5"},"uses_additional_data":false},{"leaderboard":"/sota/pose-estimation-on-crowdpose","task":"Pose Estimation","dataset":"CrowdPose","model":"SwinV2-B 1K-MIM","rank_in_archive_order":5,"of":12,"metrics":{"AP":"74.9"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-got-10k","task":"Visual Object Tracking","dataset":"GOT-10k","model":"SwinV2-L 1K-MIM","rank_in_archive_order":24,"of":42,"metrics":{"Average Overlap":"72.9"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-got-10k","task":"Visual Object Tracking","dataset":"GOT-10k","model":"SwinV2-B 1K-MIM","rank_in_archive_order":26,"of":42,"metrics":{"Average Overlap":"70.8"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-lasot","task":"Visual Object Tracking","dataset":"LaSOT","model":"SwinV2-L 1K-MIM","rank_in_archive_order":26,"of":46,"metrics":{"AUC":"70.7"},"uses_additional_data":false},{"leaderboard":"/sota/visual-object-tracking-on-lasot","task":"Visual Object Tracking","dataset":"LaSOT","model":"SwinV2-B 1K-MIM","rank_in_archive_order":31,"of":46,"metrics":{"AUC":"70"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2205.13543","atlas_url":"https://app.syntology.ai/?focus=2205.13543","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.13543"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SwinTransformer/MIM-Depth-Estimation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_violates":1,"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1828822ff8e88962","entry":"get_dataset","repo":"SwinTransformer/MIM-Depth-Estimation","repo_kind":"official","path":"dataset/base_dataset.py","file_url":"https://github.com/SwinTransformer/MIM-Depth-Estimation/blob/HEAD/dataset/base_dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1828822ff8e88962"}},{"code_sha256_prefix":"6907f72868217c02","entry":"get_num_layer_for_swin","repo":"SwinTransformer/MIM-Depth-Estimation","repo_kind":"official","path":"models/optimizer.py","file_url":"https://github.com/SwinTransformer/MIM-Depth-Estimation/blob/HEAD/models/optimizer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6907f72868217c02"}},{"code_sha256_prefix":"7c508037b40522af","entry":"str2bool","repo":"SwinTransformer/MIM-Depth-Estimation","repo_kind":"official","path":"configs/base_options.py","file_url":"https://github.com/SwinTransformer/MIM-Depth-Estimation/blob/HEAD/configs/base_options.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7c508037b40522af"}},{"code_sha256_prefix":"f9fd6241d935f07b","entry":"window_partition","repo":"SwinTransformer/MIM-Depth-Estimation","repo_kind":"official","path":"models/swin_transformer_v2.py","file_url":"https://github.com/SwinTransformer/MIM-Depth-Estimation/blob/HEAD/models/swin_transformer_v2.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9fd6241d935f07b"}},{"code_sha256_prefix":"fb32094c6dbece71","entry":"window_reverse","repo":"SwinTransformer/MIM-Depth-Estimation","repo_kind":"official","path":"models/swin_transformer_v2.py","file_url":"https://github.com/SwinTransformer/MIM-Depth-Estimation/blob/HEAD/models/swin_transformer_v2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb32094c6dbece71"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}