{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/contrastive-learning-rivals-masked-image","title":"Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation","arxiv_id":"2205.14141","date":"2022-05-27","proceeding":null,"authors":["Yixuan Wei","Han Hu","Zhenda Xie","Zheng Zhang","Yue Cao","Jianmin Bao","Dong Chen","Baining Guo"],"abstract":"Masked image modeling (MIM) learns representations with remarkably good fine-tuning performances, overshadowing previous prevalent pre-training approaches such as image classification, instance contrastive learning, and image-text alignment. In this paper, we show that the inferior fine-tuning performance of these pre-training approaches can be significantly improved by a simple post-processing in the form of feature distillation (FD). The feature distillation converts the old representations to new representations that have a few desirable properties just like those representations produced by MIM. These properties, which we aggregately refer to as optimization friendliness, are identified and analyzed by a set of attention- and optimization-related diagnosis tools. With these properties, the new representations show strong fine-tuning performance. Specifically, the contrastive self-supervised learning methods are made as competitive in fine-tuning as the state-of-the-art masked image modeling (MIM) algorithms. The CLIP models' fine-tuning performance is also significantly improved, with a CLIP ViT-L model reaching 89.0% top-1 accuracy on ImageNet-1K classification. On the 3-billion-parameter SwinV2-G model, the fine-tuning accuracy is improved by +1.5 mIoU / +1.1 mAP to 61.4 mIoU / 64.2 mAP on ADE20K semantic segmentation and COCO object detection, respectively, creating new records on both benchmarks. More importantly, our work provides a way for the future research to focus more effort on the generality and scalability of the learnt representations without being pre-occupied with optimization friendliness since it can be enhanced rather easily. The code will be available at https://github.com/SwinTransformer/Feature-Distillation.","url_abs":"https://arxiv.org/abs/2205.14141v3","url_pdf":"https://arxiv.org/pdf/2205.14141v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"contrastive-learning-rivals-masked-image","repo_url":"https://github.com/SwinTransformer/Feature-Distillation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"mim","method_name":"MIM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"FD (CLIP ViT-L-336)","rank_in_archive_order":26,"of":1060,"metrics":{"Number of params":"307M","Top 1 Accuracy":"89.0%"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco","task":"Instance Segmentation","dataset":"COCO test-dev","model":"FD-SwinV2-G","rank_in_archive_order":4,"of":112,"metrics":{"mask AP":"55.4"},"uses_additional_data":true},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"FD-SwinV2-G","rank_in_archive_order":11,"of":225,"metrics":{"box mAP":"64.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"FD-SwinV2-G","rank_in_archive_order":9,"of":235,"metrics":{"Params (M)":"3000","Validation mIoU":"61.4"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k-val","task":"Semantic Segmentation","dataset":"ADE20K val","model":"FD-SwinV2-G","rank_in_archive_order":4,"of":95,"metrics":{"mIoU":"61.4"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2205.14141","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2205.14141"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/SwinTransformer/Feature-Distillation","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":1,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"144d10b49baeb8a6","entry":"window_partition","repo":"SwinTransformer/Feature-Distillation","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/SwinTransformer/Feature-Distillation/blob/HEAD/models/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"144d10b49baeb8a6"}},{"code_sha256_prefix":"61bf152e6a42a184","entry":"window_reverse","repo":"SwinTransformer/Feature-Distillation","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/SwinTransformer/Feature-Distillation/blob/HEAD/models/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"61bf152e6a42a184"}},{"code_sha256_prefix":"6fb8740c47c862be","entry":"build_model","repo":"SwinTransformer/Feature-Distillation","repo_kind":"official","path":"models/build.py","file_url":"https://github.com/SwinTransformer/Feature-Distillation/blob/HEAD/models/build.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"6fb8740c47c862be"}},{"code_sha256_prefix":"3426eb5107695fae","entry":"build_scheduler","repo":"SwinTransformer/Feature-Distillation","repo_kind":"official","path":"lr_scheduler.py","file_url":"https://github.com/SwinTransformer/Feature-Distillation/blob/HEAD/lr_scheduler.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"3426eb5107695fae"}},{"code_sha256_prefix":"1de8c8d77d0d6d9f","entry":"build_swin","repo":"SwinTransformer/Feature-Distillation","repo_kind":"official","path":"models/swin_transformer.py","file_url":"https://github.com/SwinTransformer/Feature-Distillation/blob/HEAD/models/swin_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"1de8c8d77d0d6d9f"}},{"code_sha256_prefix":"80450600f7f09b0a","entry":"create_logger","repo":"SwinTransformer/Feature-Distillation","repo_kind":"official","path":"logger.py","file_url":"https://github.com/SwinTransformer/Feature-Distillation/blob/HEAD/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"80450600f7f09b0a"}},{"code_sha256_prefix":"6d0a4fffacbeaf60","entry":"get_config","repo":"SwinTransformer/Feature-Distillation","repo_kind":"official","path":"config.py","file_url":"https://github.com/SwinTransformer/Feature-Distillation/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"6d0a4fffacbeaf60"}},{"code_sha256_prefix":"02d20187a86b6a06","entry":"update_config","repo":"SwinTransformer/Feature-Distillation","repo_kind":"official","path":"config.py","file_url":"https://github.com/SwinTransformer/Feature-Distillation/blob/HEAD/config.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"02d20187a86b6a06"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}