{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unmasked-teacher-towards-training-efficient","title":"Unmasked Teacher: Towards Training-Efficient Video Foundation Models","arxiv_id":"2303.16058","date":"2023-03-28","proceeding":"ICCV 2023 1","authors":["Kunchang Li","Yali Wang","Yizhuo Li","Yi Wang","Yinan He","LiMin Wang","Yu Qiao"],"abstract":"Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multimodal friendliness. With a progressive pre-training framework, our model can handle various tasks including scene-related, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks. The code and models will be released at https://github.com/OpenGVLab/unmasked_teacher.","url_abs":"https://arxiv.org/abs/2303.16058v2","url_pdf":"https://arxiv.org/pdf/2303.16058v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unmasked-teacher-towards-training-efficient","repo_url":"https://github.com/opengvlab/unmasked_teacher","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"spatio-temporal-action-localization","task_name":"Spatio-Temporal Action Localization"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"},{"task_slug":"zero-shot-video-retrieval","task_name":"Zero-Shot Video Retrieval"},{"task_slug":"cross-modal-alignment","task_name":"cross-modal alignment"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"Unmasked Teacher (ViT-L)","rank_in_archive_order":8,"of":207,"metrics":{"Acc@1":"90.6","Acc@5":"98.7","FLOPs (G) x views":"1434×3×4","Parameters (M)":"304"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"UMT-L (ViT-L/16)","rank_in_archive_order":9,"of":207,"metrics":{"Acc@1":"90.6","Acc@5":"98.7"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"UMT-L (ViT-L/16)","rank_in_archive_order":8,"of":65,"metrics":{"Top-1 Accuracy":"90.5","Top-5 Accuracy":"98.8"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-kinetics-700","task":"Action Classification","dataset":"Kinetics-700","model":"UMT-L (ViT-L/16)","rank_in_archive_order":5,"of":36,"metrics":{"Top-1 Accuracy":"83.6","Top-5 Accuracy":"96.7"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-moments-in-time","task":"Action Classification","dataset":"MiT","model":"UMT-L (ViT-L/16)","rank_in_archive_order":3,"of":29,"metrics":{"Top 1 Accuracy":"48.7","Top 5 Accuracy":"78.2"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"UMT-L (ViT-L/16)","rank_in_archive_order":9,"of":38,"metrics":{"mAP":"39.8"},"uses_additional_data":true},{"leaderboard":"/sota/video-question-answering-on-activitynet-qa","task":"Video Question Answering","dataset":"ActivityNet-QA","model":"UMT-L (ViT-L/16)","rank_in_archive_order":10,"of":36,"metrics":{"Accuracy":"47.9"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"UMT-L (ViT-L/16)","rank_in_archive_order":6,"of":31,"metrics":{"text-to-video R@1":"66.8","text-to-video R@10":"94.9","text-to-video R@5":"89.1","video-to-text R@1":"64.4","video-to-text R@10":"94.8","video-to-text R@5":"89.1"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"UMT-L (ViT-L/16)","rank_in_archive_order":5,"of":40,"metrics":{"text-to-video R@1":"70.4","text-to-video R@10":"93.5","text-to-video R@5":"90.1","video-to-text R@1":"65.7","video-to-text R@10":"93.3","video-to-text R@5":"89.6"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-lsmdc","task":"Video Retrieval","dataset":"LSMDC","model":"UMT-L (ViT-L/16)","rank_in_archive_order":3,"of":38,"metrics":{"text-to-video R@1":"43.0","text-to-video R@10":"73.0","text-to-video R@5":"65.5","video-to-text R@1":"41.4","video-to-text R@10":"71.5","video-to-text R@5":"64.3"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt","task":"Video Retrieval","dataset":"MSR-VTT","model":"UMT-L (ViT-L/16)","rank_in_archive_order":5,"of":40,"metrics":{"text-to-video R@1":"58.8","text-to-video R@10":"87.1","text-to-video R@5":"81.0","video-to-text R@1":"58.6","video-to-text R@10":"86.5","video-to-text R@5":"81.6"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-ssv2-label-retrieval","task":"Video Retrieval","dataset":"SSv2-label retrieval","model":"UMT-L (ViT-L/16)","rank_in_archive_order":1,"of":5,"metrics":{"text-to-video R@1":"73.3","text-to-video R@10":"96.6","text-to-video R@5":"92.7"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-ssv2-template-retrieval","task":"Video Retrieval","dataset":"SSv2-template retrieval","model":"UMT-L (ViT-L/16)","rank_in_archive_order":1,"of":5,"metrics":{"text-to-video R@1":"90.8","text-to-video R@10":"100.0","text-to-video R@5":"100.0"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-vatex","task":"Video Retrieval","dataset":"VATEX","model":"Unmasked Teacher","rank_in_archive_order":5,"of":13,"metrics":{"text-to-video R@1":"72","text-to-video R@10":"97.8","text-to-video R@5":"95.1","video-to-text R@1":"86.0","video-to-text R@10":"99.6"},"uses_additional_data":false},{"leaderboard":"/sota/visual-question-answering-on-msrvtt-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSRVTT-QA","model":"UMT-L (ViT-L/16)","rank_in_archive_order":7,"of":34,"metrics":{"Accuracy":"0.471"},"uses_additional_data":true},{"leaderboard":"/sota/visual-question-answering-on-msvd-qa-1","task":"Visual Question Answering (VQA)","dataset":"MSVD-QA","model":"UMT-L (ViT-L/16)","rank_in_archive_order":13,"of":36,"metrics":{"Accuracy":"0.552"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-activitynet","task":"Zero-Shot Video Retrieval","dataset":"ActivityNet","model":"UMT-L (ViT-L/16)","rank_in_archive_order":4,"of":12,"metrics":{"text-to-video R@1":"42.8","text-to-video R@10":"79.8","text-to-video R@5":"69.6","video-to-text R@1":"40.7","video-to-text R@10":"78.6","video-to-text R@5":"67.6"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-didemo","task":"Zero-Shot Video Retrieval","dataset":"DiDeMo","model":"UMT-L (ViT-L/16)","rank_in_archive_order":6,"of":26,"metrics":{"text-to-video R@1":"48.6","text-to-video R@10":"79.0","text-to-video R@5":"72.9","video-to-text R@1":"49.9","video-to-text R@10":"81.4","video-to-text R@5":"74.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-lsmdc","task":"Zero-Shot Video Retrieval","dataset":"LSMDC","model":"UMT-L (ViT-L/16)","rank_in_archive_order":4,"of":16,"metrics":{"text-to-video R@1":"25.2","text-to-video R@10":"50.5","text-to-video R@5":"43.0","video-to-text R@1":"23.2","video-to-text R@10":" 44.2","video-to-text R@5":"37.7"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msr-vtt","task":"Zero-Shot Video Retrieval","dataset":"MSR-VTT","model":"UMT-L (ViT-L/16)","rank_in_archive_order":11,"of":41,"metrics":{"text-to-video R@1":"42.6","text-to-video R@10":"73.1","text-to-video R@5":"64.4","video-to-text R@1":"38.6","video-to-text R@10":"69.6","video-to-text R@5":"59.8"},"uses_additional_data":true},{"leaderboard":"/sota/zero-shot-video-retrieval-on-msvd","task":"Zero-Shot Video Retrieval","dataset":"MSVD","model":"UMT-L (ViT-L/16)","rank_in_archive_order":7,"of":14,"metrics":{"text-to-video R@1":"49.0","text-to-video R@10":"84.7","text-to-video R@5":"76.9","video-to-text R@1":"74.5","video-to-text R@10":"92.8","video-to-text R@5":"89.7"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2303.16058","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.16058"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/unmasked_teacher","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":1,"ran_fixture":1,"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":8,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"da651e3979a18f84","entry":"get_sinusoid_encoding_table","repo":"opengvlab/unmasked_teacher","repo_kind":"official","path":"single_modality/action_detection/modeling_finetune.py","file_url":"https://github.com/opengvlab/unmasked_teacher/blob/HEAD/single_modality/action_detection/modeling_finetune.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da651e3979a18f84"}},{"code_sha256_prefix":"f64c56c357505158","entry":"interpolate_temporal_pos_embed","repo":"opengvlab/unmasked_teacher","repo_kind":"official","path":"multi_modality/models/utils.py","file_url":"https://github.com/opengvlab/unmasked_teacher/blob/HEAD/multi_modality/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f64c56c357505158"}},{"code_sha256_prefix":"5349fd1148712a53","entry":"load_temp_embed_with_mismatch","repo":"opengvlab/unmasked_teacher","repo_kind":"official","path":"multi_modality/models/utils.py","file_url":"https://github.com/opengvlab/unmasked_teacher/blob/HEAD/multi_modality/models/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5349fd1148712a53"}},{"code_sha256_prefix":"604766e7f8e055b3","entry":"RandomMaskingGenerator","repo":"opengvlab/unmasked_teacher","repo_kind":"official","path":"multi_modality/models/mask.py","file_url":"https://github.com/opengvlab/unmasked_teacher/blob/HEAD/multi_modality/models/mask.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"604766e7f8e055b3"}},{"code_sha256_prefix":"4858eefb54696701","entry":"TubeMaskingGenerator","repo":"opengvlab/unmasked_teacher","repo_kind":"official","path":"multi_modality/models/mask.py","file_url":"https://github.com/opengvlab/unmasked_teacher/blob/HEAD/multi_modality/models/mask.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4858eefb54696701"}},{"code_sha256_prefix":"29416043c7035e7c","entry":"get_loss_scale_for_deepspeed","repo":"opengvlab/unmasked_teacher","repo_kind":"official","path":"single_modality/action_detection/engine_for_finetuning.py","file_url":"https://github.com/opengvlab/unmasked_teacher/blob/HEAD/single_modality/action_detection/engine_for_finetuning.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"29416043c7035e7c"}},{"code_sha256_prefix":"431de8c91d7b558f","entry":"get_sim","repo":"opengvlab/unmasked_teacher","repo_kind":"official","path":"multi_modality/models/criterions.py","file_url":"https://github.com/opengvlab/unmasked_teacher/blob/HEAD/multi_modality/models/criterions.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"431de8c91d7b558f"}},{"code_sha256_prefix":"dbd049b02a63c3a1","entry":"interpolate_pos_embed","repo":"opengvlab/unmasked_teacher","repo_kind":"official","path":"multi_modality/models/utils.py","file_url":"https://github.com/opengvlab/unmasked_teacher/blob/HEAD/multi_modality/models/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbd049b02a63c3a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}