{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vlm-task-agnostic-video-language-model-pre","title":"VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding","arxiv_id":"2105.09996","date":"2021-05-20","proceeding":"Findings (ACL) 2021 8","authors":["Hu Xu","Gargi Ghosh","Po-Yao Huang","Prahal Arora","Masoumeh Aminzadeh","Christoph Feichtenhofer","Florian Metze","Luke Zettlemoyer"],"abstract":"We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks. Existing pre-training are task-specific by adopting either a single cross-modal encoder that requires both modalities, limiting their use for retrieval-style end tasks or more complex multitask learning with two unimodal encoders, limiting early cross-modal fusion. We instead introduce new pretraining masking schemes that better mix across modalities (e.g. by forcing masks for text to predict the closest video embeddings) while also maintaining separability (e.g. unimodal predictions are sometimes required, without using all the input). Experimental results show strong performance across a wider range of tasks than any previous methods, often outperforming task-specific pre-training. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.","url_abs":"https://arxiv.org/abs/2105.09996v3","url_pdf":"https://arxiv.org/pdf/2105.09996v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vlm-task-agnostic-video-language-model-pre","repo_url":"https://github.com/pytorch/fairseq","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-segmentation","task_name":"Action Segmentation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"video-captioning","task_name":"Video Captioning"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-segmentation-on-coin","task":"Action Segmentation","dataset":"COIN","model":"VLM","rank_in_archive_order":5,"of":9,"metrics":{"Frame accuracy":"68.4"},"uses_additional_data":true},{"leaderboard":"/sota/temporal-action-localization-on-crosstask","task":"Temporal Action Localization","dataset":"CrossTask","model":"VLM","rank_in_archive_order":2,"of":7,"metrics":{"Recall":"46.5"},"uses_additional_data":true},{"leaderboard":"/sota/video-captioning-on-youcook2","task":"Video Captioning","dataset":"YouCook2","model":"VLM","rank_in_archive_order":5,"of":14,"metrics":{"BLEU-3":"17.78","BLEU-4":"12.27","CIDEr":"1.3869","METEOR":"18.22","ROUGE-L":"41.51"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-msr-vtt-1ka","task":"Video Retrieval","dataset":"MSR-VTT-1kA","model":"VLM","rank_in_archive_order":52,"of":63,"metrics":{"text-to-video Median Rank":"4","text-to-video R@1":"28.10","text-to-video R@10":"67.40","text-to-video R@5":"55.50"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-youcook2","task":"Video Retrieval","dataset":"YouCook2","model":"VLM","rank_in_archive_order":7,"of":16,"metrics":{"text-to-video Median Rank":"4","text-to-video R@1":"27.05","text-to-video R@10":"69.38","text-to-video R@5":"56.88"},"uses_additional_data":true}],"syntology":{"syntology_url":"https://syntology.ai/paper/2105.09996","atlas_url":"https://app.syntology.ai/?focus=2105.09996","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}