{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/videomae-v2-scaling-video-masked-autoencoders","title":"VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking","arxiv_id":"2303.16727","date":"2023-03-29","proceeding":"CVPR 2023 1","authors":["LiMin Wang","Bingkun Huang","Zhiyu Zhao","Zhan Tong","Yinan He","Yi Wang","Yali Wang","Yu Qiao"],"abstract":"Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder can still further reduce the overall computational cost. This enables the efficient pre-training of billion-level models in video. We also use a progressive training paradigm that involves an initial pre-training on a diverse multi-sourced unlabeled dataset, followed by a post-pre-training on a mixed labeled dataset. Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90.0% on K400 and 89.9% on K600) and Something-Something (68.7% on V1 and 77.0% on V2). In addition, we extensively verify the pre-trained video ViT models on a variety of downstream tasks, demonstrating its effectiveness as a general video representation learner. The code and model is available at \\url{https://github.com/OpenGVLab/VideoMAEv2}.","url_abs":"https://arxiv.org/abs/2303.16727v2","url_pdf":"https://arxiv.org/pdf/2303.16727v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"videomae-v2-scaling-video-masked-autoencoders","repo_url":"https://github.com/OpenGVLab/VideoMAEv2","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"action-recognition-in-videos-2","task_name":"Action Recognition In Videos"},{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"self-supervised-action-recognition","task_name":"Self-Supervised Action Recognition"},{"task_slug":"spatio-temporal-action-localization","task_name":"Spatio-Temporal Action Localization"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"VideoMAE V2-g (64x266x266)","rank_in_archive_order":12,"of":207,"metrics":{"Acc@1":"90.0","Acc@5":"98.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"VideoMAE V2-g","rank_in_archive_order":25,"of":207,"metrics":{"Acc@1":"88.5","Acc@5":"98.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"VideoMAE V2-g (64x266x266)","rank_in_archive_order":11,"of":65,"metrics":{"Top-1 Accuracy":"89.9","Top-5 Accuracy":"98.5"},"uses_additional_data":true},{"leaderboard":"/sota/action-classification-on-kinetics-600","task":"Action Classification","dataset":"Kinetics-600","model":"VideoMAE V2-g","rank_in_archive_order":17,"of":65,"metrics":{"Top-1 Accuracy":"88.8","Top-5 Accuracy":"98.2"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"VideoMAE V2-g","rank_in_archive_order":3,"of":38,"metrics":{"mAP":"42.6"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-hmdb-51","task":"Action Recognition","dataset":"HMDB-51","model":"VideoMAE V2-g","rank_in_archive_order":1,"of":77,"metrics":{"Average accuracy of 3 splits":"88.7"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something-1","task":"Action Recognition","dataset":"Something-Something V1","model":"VideoMAE V2-g","rank_in_archive_order":2,"of":74,"metrics":{"Top 1 Accuracy":"68.7","Top 5 Accuracy":"91.9"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"VideoMAE V2-g","rank_in_archive_order":5,"of":123,"metrics":{"GFLOPs":"2544x6","Parameters":"1013","Top-1 Accuracy":"77.0","Top-5 Accuracy":"95.9"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-ucf101","task":"Action Recognition","dataset":"UCF101","model":"VideoMAE V2-g","rank_in_archive_order":2,"of":91,"metrics":{"3-fold Accuracy":"99.6"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-ava-v2-2","task":"Action Recognition In Videos","dataset":"AVA v2.2","model":"VideoMAE V2","rank_in_archive_order":2,"of":2,"metrics":{"mAP (Val)":"18.24"},"uses_additional_data":false},{"leaderboard":"/sota/self-supervised-action-recognition-on-ucf101","task":"Self-Supervised Action Recognition","dataset":"UCF101","model":"VideoMAE V2-g","rank_in_archive_order":1,"of":53,"metrics":{"3-fold Accuracy":"99.6"},"uses_additional_data":true},{"leaderboard":"/sota/spatio-temporal-action-localization-on-ava","task":"Spatio-Temporal Action Localization","dataset":"AVA-Kinetics","model":"VideoMAE V2-g","rank_in_archive_order":1,"of":7,"metrics":{"val mAP":"42.6"},"uses_additional_data":true},{"leaderboard":"/sota/temporal-action-localization-on-fineaction","task":"Temporal Action Localization","dataset":"FineAction","model":"VideoMAE V2-g","rank_in_archive_order":5,"of":9,"metrics":{"mAP":"18.24","mAP IOU@0.5":"29.07","mAP IOU@0.75":"17.66","mAP IOU@0.95":"5.07"},"uses_additional_data":true},{"leaderboard":"/sota/temporal-action-localization-on-thumos14","task":"Temporal Action Localization","dataset":"THUMOS’14","model":"ActionFormer (VideoMAE V2-g features)","rank_in_archive_order":8,"of":42,"metrics":{"Avg mAP (0.3:0.7)":"69.6","mAP IOU@0.3":"84.0","mAP IOU@0.4":"79.6","mAP IOU@0.5":"73.0","mAP IOU@0.6":"63.5","mAP IOU@0.7":"47.7"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2303.16727","atlas_url":"https://app.syntology.ai/?focus=2303.16727","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2303.16727"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OpenGVLab/VideoMAEv2","reach":null}],"summary":{"ran":2,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"81776e1d67d5b65d","entry":"CosAttention","repo":"OpenGVLab/VideoMAEv2","repo_kind":"official","path":"models/modeling_pretrain.py","file_url":"https://github.com/OpenGVLab/VideoMAEv2/blob/HEAD/models/modeling_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81776e1d67d5b65d"}},{"code_sha256_prefix":"8ccae2733dda5cc9","entry":"PatchEmbed","repo":"OpenGVLab/VideoMAEv2","repo_kind":"official","path":"models/modeling_pretrain.py","file_url":"https://github.com/OpenGVLab/VideoMAEv2/blob/HEAD/models/modeling_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8ccae2733dda5cc9"}},{"code_sha256_prefix":"ef3f731f47fa62cd","entry":"Block","repo":"OpenGVLab/VideoMAEv2","repo_kind":"official","path":"models/modeling_pretrain.py","file_url":"https://github.com/OpenGVLab/VideoMAEv2/blob/HEAD/models/modeling_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef3f731f47fa62cd"}},{"code_sha256_prefix":"77fa748016cdd5bb","entry":"PretrainVisionTransformer","repo":"OpenGVLab/VideoMAEv2","repo_kind":"official","path":"models/modeling_pretrain.py","file_url":"https://github.com/OpenGVLab/VideoMAEv2/blob/HEAD/models/modeling_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77fa748016cdd5bb"}},{"code_sha256_prefix":"23510dbbfb59c729","entry":"PretrainVisionTransformerDecoder","repo":"OpenGVLab/VideoMAEv2","repo_kind":"official","path":"models/modeling_pretrain.py","file_url":"https://github.com/OpenGVLab/VideoMAEv2/blob/HEAD/models/modeling_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23510dbbfb59c729"}},{"code_sha256_prefix":"a7e14063cbbf7bfa","entry":"PretrainVisionTransformerEncoder","repo":"OpenGVLab/VideoMAEv2","repo_kind":"official","path":"models/modeling_pretrain.py","file_url":"https://github.com/OpenGVLab/VideoMAEv2/blob/HEAD/models/modeling_pretrain.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7e14063cbbf7bfa"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}