{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vimpac-video-pre-training-via-masked-token","title":"VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning","arxiv_id":"2106.11250","date":"2021-06-21","proceeding":null,"authors":["Hao Tan","Jie Lei","Thomas Wolf","Mohit Bansal"],"abstract":"Video understanding relies on perceiving the global content and modeling its internal connections (e.g., causality, movement, and spatio-temporal correspondence). To learn these interactions, we apply a mask-then-predict pre-training task on discretized video tokens generated via VQ-VAE. Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e.g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations. To deal with this issue, we propose a block-wise masking strategy where we mask neighboring video tokens in both spatial and temporal domains. We also add an augmentation-free contrastive learning method to further capture the global content by predicting whether the video clips are sampled from the same video. We pre-train our model on uncurated videos and show that our pre-trained model can reach state-of-the-art results on several video understanding datasets (e.g., SSV2, Diving48). Lastly, we provide detailed analyses on model scalability and pre-training method design. Code is released at https://github.com/airsplay/vimpac.","url_abs":"https://arxiv.org/abs/2106.11250v1","url_pdf":"https://arxiv.org/pdf/2106.11250v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vimpac-video-pre-training-via-masked-token","repo_url":"https://github.com/airsplay/vimpac","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-classification","task_name":"Action Classification"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"},{"method_slug":"vq-vae","method_name":"VQ-VAE"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-classification-on-kinetics-400","task":"Action Classification","dataset":"Kinetics-400","model":"VIMPAC","rank_in_archive_order":142,"of":207,"metrics":{"Acc@1":"77.4"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-on-diving-48","task":"Action Recognition","dataset":"Diving-48","model":"VIMPAC","rank_in_archive_order":11,"of":18,"metrics":{"Accuracy":"85.5"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-hmdb-51","task":"Action Recognition","dataset":"HMDB-51","model":"VIMPAC","rank_in_archive_order":61,"of":77,"metrics":{"Average accuracy of 3 splits":"65.9"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-in-videos-on-something","task":"Action Recognition","dataset":"Something-Something V2","model":"VIMPAC","rank_in_archive_order":55,"of":123,"metrics":{"Top-1 Accuracy":"68.1"},"uses_additional_data":true},{"leaderboard":"/sota/action-recognition-in-videos-on-ucf101","task":"Action Recognition","dataset":"UCF101","model":"VIMPAC","rank_in_archive_order":62,"of":91,"metrics":{"3-fold Accuracy":"92.7"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2106.11250","atlas_url":"https://app.syntology.ai/?focus=2106.11250","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.11250"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/airsplay/vimpac","reach":null}],"summary":{"ran_violates":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"708ca1cba9f2908f","entry":"is_port_in_use","repo":"airsplay/vimpac","repo_kind":"official","path":"vimpac/pretrain.py","file_url":"https://github.com/airsplay/vimpac/blob/HEAD/vimpac/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"708ca1cba9f2908f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}