Papers › Scaling Vision Transformers to 22 Billion Parameters

Scaling Vision Transformers to 22 Billion Parameters

10 Feb 2023arXiv:2302.05442archive 2025-07-28

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučić, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, Neil Houlsby

The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for "LLM-like" scaling in vision, and provides key steps towards getting there.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

lucidrains/flash-cosine-sim-attention mentioned on GitHubpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Self-attention Does Not Need O(n²) Memory (arXiv:2112.05682; that paper's own run record: 33 ran (of which 0 constructed an object rather than computing a result; 21 with no instrument failure: 3 honoured, 6 violated, 12 with no contract checked; 12 where Syntology's instrument failed) · 3 unverified) and FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv:2205.14135; that paper's own run record: 24 ran (of which 2 constructed an object rather than computing a result; 18 with no instrument failure: 0 honoured, 1 violated, 17 with no contract checked; 6 where Syntology's instrument failed) · 6 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action ClassificationFairnessImage ClassificationLinear-Probe ClassificationObject RecognitionZero-Shot Transfer Image Classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Action Classification Kinetics-400 ViT-22B Acc@1 88.0 #29 of 207 Archive leaderboard report
Image Classification ImageNet ViT-L/16 (384res, distilled from ViT-22B) Number of params 307M #15 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-L/16 (384res, distilled from ViT-22B) Top 1 Accuracy 89.6% #15 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-B/16 Number of params 86M #36 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-B/16 Top 1 Accuracy 88.6% #36 of 1060 Archive leaderboard report
Object Recognition shape bias ViT-22B-384 shape bias 86.4 #4 of 18 Archive leaderboard report
Object Recognition shape bias ViT-22B-560 shape bias 83.8 #5 of 18 Archive leaderboard report
Object Recognition shape bias ViT-22B-224 shape bias 78.0 #7 of 18 Archive leaderboard report
Zero-Shot Transfer Image Classification ImageNet LiT-22B Accuracy (Private) 85.9 #4 of 23 Archive leaderboard report
Zero-Shot Transfer Image Classification ImageNet V2 LiT-22B Accuracy (Private) 80.9 #2 of 13 Archive leaderboard report
Zero-Shot Transfer Image Classification ImageNet-A LiT-22B Accuracy (Private) 90.1 #2 of 13 Archive leaderboard report
Zero-Shot Transfer Image Classification ImageNet-R LiT-22B Accuracy 96.0 #4 of 12 Archive leaderboard report
Zero-Shot Transfer Image Classification ObjectNet LiT-22B Accuracy (Private) 87.6 #1 of 9 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections