Papers › UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer

UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer

22 Sep 2022ICLR2023 submitted 2022 9archive 2025-07-28

Anonymous

Learning discriminative spatiotemporal representation is the key problem of video understanding. Recently, Vision Transformers (ViTs) have shown their power in learning long-term video dependency with self-attention. Unfortunately, they exhibit limitations in tackling local video redundancy, due to the blind global comparison among tokens. UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format. However, this model has to require a tiresome and complicated image-pretraining phrase, before being finetuned on videos. This blocks its wide usage in practice. On the contrary, open-sourced ViTs are readily available and well-pretrained with rich image supervision. Based on these observations, we propose a generic paradigm to build a powerful family of video networks, by arming the pretrained ViTs with efficient UniFormer designs. We call this family UniFormerV2, since it inherits the concise style of the UniFormer block. But it contains brand-new local and global relation aggregators, which allow for preferable accuracy-computation balance by seamlessly integrating advantages from both ViTs and UniFormer. Without any bells and whistles, our UniFormerV2 gets the state-of-the-art recognition performance on 8 popular video benchmarks, including scene-related Kinetics-400/600/700 and Moments in Time, temporal-related Something-Something V1/V2, untrimmed ActivityNet and HACS. In particular, it is the first model to achieve 90% top-1 accuracy on Kinetics-400, to our best knowledge. The models will be released afterward.

PaperPDFCode

Syntology This paper has no arXiv id in the archive, and this site joins papers to Syntology's graph by arXiv id, so this page does not say whether the graph holds it.

Code

OpenGVLab/UniFormerV2 pytorchApache-2.0Syntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer (arXiv:2211.09552; that paper's own run record: 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 2 unverified); nothing checks that they implement this paper's method. report
innat/UniFormerV2 tfApache-2.0Syntology: not harvested report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action ClassificationAction RecognitionVideo Understanding

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Action Classification ActivityNet UniFormerV2-L Top 1 Accuracy 94.7 #1 of 1 Archive leaderboard report
Action Classification ActivityNet UniFormerV2-L Top 5 Accuracy 99.5 #1 of 1 Archive leaderboard report
Action Classification Kinetics-400 UniFormerV2-L (ViT-L, 336) Acc@1 90.0 #11 of 207 Archive leaderboard report
Action Classification Kinetics-400 UniFormerV2-L (ViT-L, 336) Acc@5 98.4 #11 of 207 Archive leaderboard report
Action Classification Kinetics-400 UniFormerV2-L (ViT-L, 336) FLOPs (G) x views 75300x3x2 #11 of 207 Archive leaderboard report
Action Classification Kinetics-400 UniFormerV2-L (ViT-L, 336) Parameters (M) 354 #11 of 207 Archive leaderboard report
Action Classification Kinetics-600 UniFormerV2-L Top-1 Accuracy 90.1 #10 of 65 Archive leaderboard report
Action Classification Kinetics-600 UniFormerV2-L Top-5 Accuracy 98.5 #10 of 65 Archive leaderboard report
Action Classification Kinetics-700 UniFormerV2-L Top-1 Accuracy 82.7 #8 of 36 Archive leaderboard report
Action Classification Kinetics-700 UniFormerV2-L Top-5 Accuracy 96.2 #8 of 36 Archive leaderboard report
Action Classification MiT UniFormerV2-L Top 1 Accuracy 47.8 #4 of 29 Archive leaderboard report
Action Classification MiT UniFormerV2-L Top 5 Accuracy 76.9 #4 of 29 Archive leaderboard report
Action Recognition HACS UniFormerV2-L Top 1 Accuracy 95.5 #2 of 8 Archive leaderboard report
Action Recognition HACS UniFormerV2-L Top 5 Accuracy 99.8 #2 of 8 Archive leaderboard report
Action Recognition Something-Something V1 UniFormerV2-L Top 1 Accuracy 62.7 #7 of 74 Archive leaderboard report
Action Recognition Something-Something V1 UniFormerV2-L Top 5 Accuracy 88.0 #7 of 74 Archive leaderboard report
Action Recognition Something-Something V2 UniFormerV2-L GFLOPs 5154 #23 of 123 Archive leaderboard report
Action Recognition Something-Something V2 UniFormerV2-L Top-1 Accuracy 73.0 #23 of 123 Archive leaderboard report
Action Recognition Something-Something V2 UniFormerV2-L Top-5 Accuracy 94.5 #23 of 123 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

AdamAttentionDense ConnectionsDropoutLayer NormalizationLinear LayerMulti-Head AttentionPosition-Wise Feed-Forward LayerResidual ConnectionSoftmaxTransformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections