Papers › Multimodal Pretraining for Dense Video Captioning
Multimodal Pretraining for Dense Video Captioning
Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, Radu Soricut
Learning specific hands-on skills such as cooking, car maintenance, and home repairs increasingly happens via instructional videos. The user experience with such videos is known to be improved by meta-information such as time-stamped annotations for the main steps involved. Generating such annotations automatically is challenging, and we describe here two relevant contributions. First, we construct and release a new dense video captioning dataset, Video Timeline Tags (ViTT), featuring a variety of instructional videos together with time-stamped annotations. Second, we explore several multimodal sequence-to-sequence pretraining strategies that leverage large unsupervised datasets of videos and caption-like texts. We pretrain and subsequently finetune dense video captioning models using both YouCook2 and ViTT. We show that such models generalize well and are robust over a wide variety of instructional videos.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Datasets
Introduced by this paper, per the archive.
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Dense Video Captioning | YouCook2 | E2vidD6-MASSalign-BiD | ROUGE-L | 39.03 | #7 of 7 | Archive leaderboard | report |
| Video Captioning | YouCook2 | E2vidD6-MASSvid-BiD | BLEU-4 | 12.04 | #6 of 14 | Archive leaderboard | report |
| Video Captioning | YouCook2 | E2vidD6-MASSvid-BiD | CIDEr | 1.22 | #6 of 14 | Archive leaderboard | report |
| Video Captioning | YouCook2 | E2vidD6-MASSvid-BiD | METEOR | 18.32 | #6 of 14 | Archive leaderboard | report |
| Video Captioning | YouCook2 | E2vidD6-MASSvid-BiD | ROUGE-L | 39.03 | #6 of 14 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections