{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exploring-target-representations-for-masked","title":"Exploring Target Representations for Masked Autoencoders","arxiv_id":"2209.03917","date":"2022-09-08","proceeding":null,"authors":["Xingbin Liu","Jinghao Zhou","Tao Kong","Xianming Lin","Rongrong Ji"],"abstract":"Masked autoencoders have become popular training paradigms for self-supervised visual representation learning. These models randomly mask a portion of the input and reconstruct the masked portion according to the target representations. In this paper, we first show that a careful choice of the target representation is unnecessary for learning good representations, since different targets tend to derive similarly behaved models. Driven by this observation, we propose a multi-stage masked distillation pipeline and use a randomly initialized model as the teacher, enabling us to effectively train high-capacity models without any efforts to carefully design target representations. Interestingly, we further explore using teachers of larger capacity, obtaining distilled students with remarkable transferring ability. On different tasks of classification, transfer learning, object detection, and semantic segmentation, the proposed method to perform masked knowledge distillation with bootstrapped teachers (dBOT) outperforms previous self-supervised methods by nontrivial margins. We hope our findings, as well as the proposed method, could motivate people to rethink the roles of target representations in pre-training masked autoencoders.The code and pre-trained models are publicly available at https://github.com/liuxingbin/dbot.","url_abs":"https://arxiv.org/abs/2209.03917v3","url_pdf":"https://arxiv.org/pdf/2209.03917v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exploring-target-representations-for-masked","repo_url":"https://github.com/liuxingbin/dbot","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"self-supervised-image-classification","task_name":"Self-Supervised Image Classification"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"dBOT ViT-H (CLIP as Teacher)","rank_in_archive_order":57,"of":1060,"metrics":{"Top 1 Accuracy":"88.2%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"dBOT ViT-L (CLIP as Teacher)","rank_in_archive_order":68,"of":1060,"metrics":{"Top 1 Accuracy":"87.8%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"dBOT ViT-B (CLIP as Teacher)","rank_in_archive_order":208,"of":1060,"metrics":{"Top 1 Accuracy":"85.7%"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco","task":"Instance Segmentation","dataset":"COCO test-dev","model":"dBOT ViT-L (CLIP)","rank_in_archive_order":26,"of":112,"metrics":{"mask AP":"48.8"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco","task":"Instance Segmentation","dataset":"COCO test-dev","model":"dBOT ViT-L","rank_in_archive_order":29,"of":112,"metrics":{"mask AP":"48.3"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco","task":"Instance Segmentation","dataset":"COCO test-dev","model":"dBOT ViT-B","rank_in_archive_order":36,"of":112,"metrics":{"mask AP":"46.3"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco","task":"Instance Segmentation","dataset":"COCO test-dev","model":"dBOT ViT-B (CLIP)","rank_in_archive_order":37,"of":112,"metrics":{"mask AP":"46.2"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"dBOT ViT-L (CLIP)","rank_in_archive_order":39,"of":225,"metrics":{"box mAP":"56.8"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"dBOT ViT-L","rank_in_archive_order":45,"of":225,"metrics":{"box mAP":"56.1"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"dBOT ViT-B (CLIP)","rank_in_archive_order":59,"of":225,"metrics":{"box mAP":"53.6"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"dBOT ViT-B","rank_in_archive_order":62,"of":225,"metrics":{"box mAP":"53.5"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-image-classification-on-1","task":"Self-Supervised Image Classification","dataset":"ImageNet (finetuned)","model":"dBOT (ViT-H/14)","rank_in_archive_order":6,"of":65,"metrics":{"Number of Params":"632M","Top 1 Accuracy":"88.0%"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"dBOT ViT-L (CLIP)","rank_in_archive_order":42,"of":235,"metrics":{"Validation mIoU":"56.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"dBOT ViT-L","rank_in_archive_order":49,"of":235,"metrics":{"Validation mIoU":"55.2"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"dBOT ViT-B (CLIP)","rank_in_archive_order":85,"of":235,"metrics":{"Validation mIoU":"52.9"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"dBOT ViT-B","rank_in_archive_order":109,"of":235,"metrics":{"Validation mIoU":"50.8"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2209.03917","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2209.03917"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/liuxingbin/dbot","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"ran_honours":1,"ran_fixture":1,"unverified":8},"by_repo_kind":{"official":{"samples":13,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c19231378e41dc5a","entry":"bool_flag","repo":"liuxingbin/dbot","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/evaluation/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c19231378e41dc5a"}},{"code_sha256_prefix":"2b5489053b222e00","entry":"clip_gradients","repo":"liuxingbin/dbot","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/evaluation/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b5489053b222e00"}},{"code_sha256_prefix":"361a6b24f11fc50a","entry":"cosine_scheduler","repo":"liuxingbin/dbot","repo_kind":"official","path":"evaluation/utils.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/evaluation/utils.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"361a6b24f11fc50a"}},{"code_sha256_prefix":"55120f2026b56aa2","entry":"drop_path","repo":"liuxingbin/dbot","repo_kind":"official","path":"evaluation/models.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/evaluation/models.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"55120f2026b56aa2"}},{"code_sha256_prefix":"a3448702d6025ce6","entry":"get_num_layer_for_vit","repo":"liuxingbin/dbot","repo_kind":"official","path":"evaluation/optim_factory.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/evaluation/optim_factory.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a3448702d6025ce6"}},{"code_sha256_prefix":"ac882d65ee949545","entry":"build_dataset","repo":"liuxingbin/dbot","repo_kind":"official","path":"evaluation/eval_cls.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/evaluation/eval_cls.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ac882d65ee949545"}},{"code_sha256_prefix":"1902796bef81ffd2","entry":"build_transform","repo":"liuxingbin/dbot","repo_kind":"official","path":"evaluation/eval_cls.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/evaluation/eval_cls.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1902796bef81ffd2"}},{"code_sha256_prefix":"336aa445b9756b05","entry":"compute_distance_matrix","repo":"liuxingbin/dbot","repo_kind":"official","path":"analysis/analysis.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/analysis/analysis.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"336aa445b9756b05"}},{"code_sha256_prefix":"b77522e9d6eaa3af","entry":"compute_mean_attention_dist","repo":"liuxingbin/dbot","repo_kind":"official","path":"analysis/analysis.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/analysis/analysis.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b77522e9d6eaa3af"}},{"code_sha256_prefix":"481536e2fedc4873","entry":"get_dataset","repo":"liuxingbin/dbot","repo_kind":"official","path":"analysis/analysis.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/analysis/analysis.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"481536e2fedc4873"}},{"code_sha256_prefix":"bb32ca015a494c58","entry":"get_parameter_groups","repo":"liuxingbin/dbot","repo_kind":"official","path":"evaluation/optim_factory.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/evaluation/optim_factory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb32ca015a494c58"}},{"code_sha256_prefix":"48bb4f68134b5f20","entry":"vit_small","repo":"liuxingbin/dbot","repo_kind":"official","path":"analysis/vision_transformer.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/analysis/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"48bb4f68134b5f20"}},{"code_sha256_prefix":"6769eebdd8702e50","entry":"vit_tiny","repo":"liuxingbin/dbot","repo_kind":"official","path":"analysis/vision_transformer.py","file_url":"https://github.com/liuxingbin/dbot/blob/HEAD/analysis/vision_transformer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6769eebdd8702e50"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}