{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tinymim-an-empirical-study-of-distilling-mim","title":"TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models","arxiv_id":"2301.01296","date":"2023-01-03","proceeding":"CVPR 2023 1","authors":["Sucheng Ren","Fangyun Wei","Zheng Zhang","Han Hu"],"abstract":"Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than CLS token- and feature-based distillation; 2) An intermediate layer of the teacher network as target perform better than that using the last layer when the depth of the student mismatches that of the teacher; 3) Weak regularization is preferred; etc. With these findings, we achieve significant fine-tuning accuracy improvements over the scratch MIM pre-training on ImageNet-1K classification, using all the ViT-Tiny, ViT-Small, and ViT-base models, with +4.2%/+2.4%/+1.4% gains, respectively. Our TinyMIM model of base size achieves 52.2 mIoU in AE20K semantic segmentation, which is +4.1 higher than the MAE baseline. Our TinyMIM model of tiny size achieves 79.6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget. This strong performance suggests an alternative way for developing small vision Transformer models, that is, by exploring better training methods rather than introducing inductive biases into architectures as in most previous works. Code is available at https://github.com/OliverRensu/TinyMIM.","url_abs":"https://arxiv.org/abs/2301.01296v1","url_pdf":"https://arxiv.org/pdf/2301.01296v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tinymim-an-empirical-study-of-distilling-mim","repo_url":"https://github.com/oliverrensu/tinymim","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"tinymim-an-empirical-study-of-distilling-mim","repo_url":"https://github.com/oliverrensu/d-igpt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"mae","method_name":"MAE"},{"method_slug":"mim","method_name":"MIM"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2301.01296","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.01296"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/oliverrensu/d-igpt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/oliverrensu/tinymim","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/OliverRensu/TinyMIM","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_honours":1,"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1},"listed":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"6fb43d8ac6c1d00c","entry":"Attention","repo":"oliverrensu/d-igpt","repo_kind":"listed","path":"DiGPT_torch/models_digpt.py","file_url":"https://github.com/oliverrensu/d-igpt/blob/HEAD/DiGPT_torch/models_digpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6fb43d8ac6c1d00c"}},{"code_sha256_prefix":"e861849da1498470","entry":"Attention","repo":"OliverRensu/TinyMIM","repo_kind":"official","path":"models_tinymim.py","file_url":"https://github.com/OliverRensu/TinyMIM/blob/HEAD/models_tinymim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e861849da1498470"}},{"code_sha256_prefix":"3def779b7964615e","entry":"Block","repo":"oliverrensu/d-igpt","repo_kind":"listed","path":"DiGPT_torch/models_digpt.py","file_url":"https://github.com/oliverrensu/d-igpt/blob/HEAD/DiGPT_torch/models_digpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3def779b7964615e"}},{"code_sha256_prefix":"01d6c22c6a63aed1","entry":"Block","repo":"OliverRensu/TinyMIM","repo_kind":"official","path":"models_tinymim.py","file_url":"https://github.com/OliverRensu/TinyMIM/blob/HEAD/models_tinymim.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"01d6c22c6a63aed1"}},{"code_sha256_prefix":"ce2a200723aec3f8","entry":"CrossAttention","repo":"oliverrensu/d-igpt","repo_kind":"listed","path":"DiGPT_torch/models_digpt.py","file_url":"https://github.com/oliverrensu/d-igpt/blob/HEAD/DiGPT_torch/models_digpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce2a200723aec3f8"}},{"code_sha256_prefix":"3d96fefa337b0261","entry":"DecoderBlock","repo":"oliverrensu/d-igpt","repo_kind":"listed","path":"DiGPT_torch/models_digpt.py","file_url":"https://github.com/oliverrensu/d-igpt/blob/HEAD/DiGPT_torch/models_digpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3d96fefa337b0261"}},{"code_sha256_prefix":"468a394a75068b28","entry":"PatchEmbed","repo":"oliverrensu/d-igpt","repo_kind":"listed","path":"DiGPT_torch/models_digpt.py","file_url":"https://github.com/oliverrensu/d-igpt/blob/HEAD/DiGPT_torch/models_digpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"468a394a75068b28"}},{"code_sha256_prefix":"5a37690a13a6157d","entry":"get_2d_sincos_pos_embed","repo":"oliverrensu/d-igpt","repo_kind":"listed","path":"DiGPT_torch/models_digpt.py","file_url":"https://github.com/oliverrensu/d-igpt/blob/HEAD/DiGPT_torch/models_digpt.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5a37690a13a6157d"}},{"code_sha256_prefix":"d86994f1871555f5","entry":"DiGPT","repo":"oliverrensu/d-igpt","repo_kind":"listed","path":"DiGPT_torch/models_digpt.py","file_url":"https://github.com/oliverrensu/d-igpt/blob/HEAD/DiGPT_torch/models_digpt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d86994f1871555f5"}},{"code_sha256_prefix":"521a3b2b4735b6d4","entry":"TinyMIMViT","repo":"OliverRensu/TinyMIM","repo_kind":"official","path":"models_tinymim.py","file_url":"https://github.com/OliverRensu/TinyMIM/blob/HEAD/models_tinymim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"521a3b2b4735b6d4"}},{"code_sha256_prefix":"2884deadbb637c3e","entry":"get_2d_sincos_pos_embed_from_grid","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"2884deadbb637c3e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}