Papers › Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions

Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions

19 Nov 2021CVPR 2022 1arXiv:2111.10337archive 2025-07-28

Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, Baining Guo

We study joint video and language (VL) pre-training to enable cross-modality learning and benefit plentiful downstream VL tasks. Existing works either extract low-quality video features or learn limited text embedding, while neglecting that high-resolution videos and diversified semantics can significantly improve cross-modality learning. In this paper, we propose a novel High-resolution and Diversified VIdeo-LAnguage pre-training model (HD-VILA) for many visual tasks. In particular, we collect a large dataset with two distinct properties: 1) the first high-resolution dataset including 371.5k hours of 720p videos, and 2) the most diversified dataset covering 15 popular YouTube categories. To enable VL pre-training, we jointly optimize the HD-VILA model by a hybrid Transformer that learns rich spatiotemporal features, and a multimodal Transformer that enforces interactions of the learned video features with diversified texts. Our pre-training model achieves new state-of-the-art results in 10 VL understanding tasks and 2 more novel text-to-visual generation tasks. For example, we outperform SOTA models with relative increases of 40.4% R@1 in zero-shot MSR-VTT text-to-video retrieval task and 55.4% in high-resolution dataset LSMDC. The learned VL embedding is also effective in generating visually pleasing and semantically relevant results in text-to-visual editing and super-resolution tasks.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

microsoft/xpretrain officialmentioned in paperpytorchNOASSERTIONSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment (arXiv:2209.06430; that paper's own run record: 4 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified) and Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning (arXiv:2210.06031; that paper's own run record: 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

RetrievalSuper-ResolutionText to Video RetrievalVideo RetrievalVocal Bursts Intensity PredictionZero-Shot Video Retrieval

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Video Retrieval ActivityNet HD-VILA text-to-video Median Rank 4 #28 of 31 Archive leaderboard report
Video Retrieval ActivityNet HD-VILA text-to-video R@1 28.5 #28 of 31 Archive leaderboard report
Video Retrieval ActivityNet HD-VILA text-to-video R@5 57.4 #28 of 31 Archive leaderboard report
Video Retrieval ActivityNet HD-VILA text-to-video R@50 94 #28 of 31 Archive leaderboard report
Video Retrieval DiDeMo HD-VILA text-to-video Median Rank 4 #37 of 40 Archive leaderboard report
Video Retrieval DiDeMo HD-VILA text-to-video R@1 28.8 #37 of 40 Archive leaderboard report
Video Retrieval DiDeMo HD-VILA text-to-video R@10 69.1 #37 of 40 Archive leaderboard report
Video Retrieval DiDeMo HD-VILA text-to-video R@5 57.4 #37 of 40 Archive leaderboard report
Video Retrieval LSMDC HD-VILA text-to-video Median Rank 15 #26 of 38 Archive leaderboard report
Video Retrieval LSMDC HD-VILA text-to-video R@1 17.4 #26 of 38 Archive leaderboard report
Video Retrieval LSMDC HD-VILA text-to-video R@10 44.1 #26 of 38 Archive leaderboard report
Video Retrieval LSMDC HD-VILA text-to-video R@5 34.1 #26 of 38 Archive leaderboard report
Video Retrieval MSR-VTT HD-VILA text-to-video MedianR 3 #17 of 40 Archive leaderboard report
Video Retrieval MSR-VTT HD-VILA text-to-video R@1 35.6 #17 of 40 Archive leaderboard report
Video Retrieval MSR-VTT HD-VILA text-to-video R@10 78 #17 of 40 Archive leaderboard report
Video Retrieval MSR-VTT HD-VILA text-to-video R@5 65.3 #17 of 40 Archive leaderboard report
Zero-Shot Video Retrieval MSR-VTT HD-VILA text-to-video Median Rank 15 #34 of 41 Archive leaderboard report
Zero-Shot Video Retrieval MSR-VTT HD-VILA text-to-video R@1 14.6 #34 of 41 Archive leaderboard report
Zero-Shot Video Retrieval MSR-VTT HD-VILA text-to-video R@10 44.1 #34 of 41 Archive leaderboard report
Zero-Shot Video Retrieval MSR-VTT HD-VILA text-to-video R@5 34.4 #34 of 41 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position EncodingsAdamAttentionBPEDense ConnectionsDropoutLabel SmoothingLayer NormalizationLinear LayerMulti-Head AttentionPosition-Wise Feed-Forward LayerResidual ConnectionSoftmaxTransformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections