Papers › VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

20 May 2021Findings (ACL) 2021 8arXiv:2105.09996archive 2025-07-28

Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer

We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks. Existing pre-training are task-specific by adopting either a single cross-modal encoder that requires both modalities, limiting their use for retrieval-style end tasks or more complex multitask learning with two unimodal encoders, limiting early cross-modal fusion. We instead introduce new pretraining masking schemes that better mix across modalities (e.g. by forcing masks for text to predict the closest video embeddings) while also maintaining separability (e.g. unimodal predictions are sometimes required, without using all the input). Experimental results show strong performance across a wider range of tasks than any previous methods, often outperforming task-specific pre-training. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

pytorch/fairseq officialmentioned in paperpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer (arXiv:2105.08206; that paper's own run record: 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result), Cascaded Head-colliding Attention (arXiv:2105.14850; that paper's own run record: 9 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 1 violated, 2 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified) and Modeling Dual Read/Write Paths for Simultaneous Machine Translation (arXiv:2203.09163; that paper's own run record: 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 5 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action SegmentationLanguage ModelingLanguage ModellingRetrievalTemporal Action LocalizationVideo CaptioningVideo RetrievalVideo Understanding

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Action Segmentation COIN VLM Frame accuracy 68.4 #5 of 9 Archive leaderboard report
Temporal Action Localization CrossTask VLM Recall 46.5 #2 of 7 Archive leaderboard report
Video Captioning YouCook2 VLM BLEU-3 17.78 #5 of 14 Archive leaderboard report
Video Captioning YouCook2 VLM BLEU-4 12.27 #5 of 14 Archive leaderboard report
Video Captioning YouCook2 VLM CIDEr 1.3869 #5 of 14 Archive leaderboard report
Video Captioning YouCook2 VLM METEOR 18.22 #5 of 14 Archive leaderboard report
Video Captioning YouCook2 VLM ROUGE-L 41.51 #5 of 14 Archive leaderboard report
Video Retrieval MSR-VTT-1kA VLM text-to-video Median Rank 4 #52 of 63 Archive leaderboard report
Video Retrieval MSR-VTT-1kA VLM text-to-video R@1 28.10 #52 of 63 Archive leaderboard report
Video Retrieval MSR-VTT-1kA VLM text-to-video R@10 67.40 #52 of 63 Archive leaderboard report
Video Retrieval MSR-VTT-1kA VLM text-to-video R@5 55.50 #52 of 63 Archive leaderboard report
Video Retrieval YouCook2 VLM text-to-video Median Rank 4 #7 of 16 Archive leaderboard report
Video Retrieval YouCook2 VLM text-to-video R@1 27.05 #7 of 16 Archive leaderboard report
Video Retrieval YouCook2 VLM text-to-video R@10 69.38 #7 of 16 Archive leaderboard report
Video Retrieval YouCook2 VLM text-to-video R@5 56.88 #7 of 16 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections