{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-vision-transformers","title":"Scaling Vision Transformers","arxiv_id":"2106.04560","date":"2021-06-08","proceeding":"CVPR 2022 1","authors":["Xiaohua Zhai","Alexander Kolesnikov","Neil Houlsby","Lucas Beyer"],"abstract":"Attention-based neural networks such as the Vision Transformer (ViT) have recently attained state-of-the-art results on many computer vision benchmarks. Scale is a primary ingredient in attaining excellent results, therefore, understanding a model's scaling properties is a key to designing future generations effectively. While the laws for scaling Transformer language models have been studied, it is unknown how Vision Transformers scale. To address this, we scale ViT models and data, both up and down, and characterize the relationships between error rate, data, and compute. Along the way, we refine the architecture and training of ViT, reducing memory consumption and increasing accuracy of the resulting models. As a result, we successfully train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90.45% top-1 accuracy. The model also performs well for few-shot transfer, for example, reaching 84.86% top-1 accuracy on ImageNet with only 10 examples per class.","url_abs":"https://arxiv.org/abs/2106.04560v2","url_pdf":"https://arxiv.org/pdf/2106.04560v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-vision-transformers","repo_url":"https://github.com/google-research/big_vision","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"few-shot-image-classification","task_name":"Few-Shot Image Classification"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[{"slug":"jft-3b","name":"JFT-3B","full_name":"JFT-3B"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet-real","task":"Image Classification","dataset":"ImageNet ReaL","model":"ViT-G/14","rank_in_archive_order":11,"of":57,"metrics":{"Accuracy":"90.81%"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-imagenet-v2","task":"Image Classification","dataset":"ImageNet V2","model":"ViT-G/14","rank_in_archive_order":6,"of":33,"metrics":{"Top 1 Accuracy":"83.33"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-objectnet","task":"Image Classification","dataset":"ObjectNet","model":"ViT-G/14","rank_in_archive_order":14,"of":106,"metrics":{"Top-1 Accuracy":"70.53"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-objectnet","task":"Image Classification","dataset":"ObjectNet","model":"NS (Eff.-L2)","rank_in_archive_order":16,"of":106,"metrics":{"Top-1 Accuracy":"68.5"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-vtab-1k-1","task":"Image Classification","dataset":"VTAB-1k","model":"ViT-G/14","rank_in_archive_order":3,"of":34,"metrics":{"Top-1 Accuracy":"78.29"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.04560","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}