</> Code with Papers by Syntology

Papers › Scaling Vision Transformers to 22 Billion Parameters

Scaling Vision Transformers to 22 Billion Parameters

10 Feb 2023arXiv:2302.05442archive 2025-07-28

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučić, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, Neil Houlsby

The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for "LLM-like" scaling in vision, and provides key steps towards getting there.

PaperPDFCode

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

lucidrains/flash-cosine-sim-attention mentioned on GitHubpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action ClassificationFairnessImage ClassificationLinear-Probe ClassificationObject RecognitionZero-Shot Transfer Image Classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Action Classification Kinetics-400 ViT-22B Acc@1 88.0 #29 of 207 Archive leaderboard report
Image Classification ImageNet ViT-L/16 (384res, distilled from ViT-22B) Number of params 307M #15 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-L/16 (384res, distilled from ViT-22B) Top 1 Accuracy 89.6% #15 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-B/16 Number of params 86M #36 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-B/16 Top 1 Accuracy 88.6% #36 of 1060 Archive leaderboard report
Object Recognition shape bias ViT-22B-384 shape bias 86.4 #4 of 18 Archive leaderboard report
Object Recognition shape bias ViT-22B-560 shape bias 83.8 #5 of 18 Archive leaderboard report
Object Recognition shape bias ViT-22B-224 shape bias 78.0 #7 of 18 Archive leaderboard report
Zero-Shot Transfer Image Classification ImageNet LiT-22B Accuracy (Private) 85.9 #4 of 23 Archive leaderboard report
Zero-Shot Transfer Image Classification ImageNet V2 LiT-22B Accuracy (Private) 80.9 #2 of 13 Archive leaderboard report
Zero-Shot Transfer Image Classification ImageNet-A LiT-22B Accuracy (Private) 90.1 #2 of 13 Archive leaderboard report
Zero-Shot Transfer Image Classification ImageNet-R LiT-22B Accuracy 96.0 #4 of 12 Archive leaderboard report
Zero-Shot Transfer Image Classification ObjectNet LiT-22B Accuracy (Private) 87.6 #1 of 9 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections