Papers › All in One: Exploring Unified Video-Language Pre-training

All in One: Exploring Unified Video-Language Pre-training

14 Mar 2022CVPR 2023 1arXiv:2203.07303archive 2025-07-28

Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, XiaoHu Qie, Mike Zheng Shou

Mainstream Video-Language Pre-training models \cite{actbert,clipbert,violet} consist of three parts, a video encoder, a text encoder, and a video-text fusion Transformer. They pursue better performance via utilizing heavier unimodal encoders or multimodal fusion Transformers, resulting in increased parameters with lower efficiency in downstream tasks. In this work, we for the first time introduce an end-to-end video-language model, namely \textit{all-in-one Transformer}, that embeds raw video and textual signals into joint representations using a unified backbone architecture. We argue that the unique temporal information of video data turns out to be a key barrier hindering the design of a modality-agnostic Transformer. To overcome the challenge, we introduce a novel and effective token rolling operation to encode temporal representations from video clips in a non-parametric manner. The careful design enables the representation learning of both video-text multimodal inputs and unimodal inputs using a unified backbone model. Our pre-trained all-in-one Transformer is transferred to various downstream video-text tasks after fine-tuning, including text-video retrieval, video-question answering, multiple choice and visual commonsense reasoning. State-of-the-art performances with the minimal model FLOPs on nine datasets demonstrate the superiority of our method compared to the competitive counterparts. The code and pretrained model have been released in https://github.com/showlab/all-in-one.

Paper PDF Conference PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

showlab/all-in-one officialmentioned in papermentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

All Language Modelling Multiple-choice Question Answering Representation Learning Retrieval TGIF-Action TGIF-Frame TGIF-Transition Video Question Answering Video Retrieval Visual Commonsense Reasoning Visual Question Answering (VQA)

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Video Question Answering	STAR Benchmark	All-in-one	Average Accuracy	47.5	#10 of 17	Archive leaderboard	report
Video Retrieval	MSR-VTT-1kA	All-in-one-B	text-to-video R@1	37.9	#42 of 63	Archive leaderboard	report
Video Retrieval	MSR-VTT-1kA	All-in-one-B	text-to-video R@10	77.1	#42 of 63	Archive leaderboard	report
Video Retrieval	MSR-VTT-1kA	All-in-one-B	text-to-video R@5	68.1	#42 of 63	Archive leaderboard	report
Visual Question Answering (VQA)	MSRVTT-QA	All-in-one-B	Accuracy	0.443	#17 of 34	Archive leaderboard	report
Visual Question Answering (VQA)	MSVD-QA	All-in-one-B	Accuracy	0.483	#24 of 36	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections