Papers › When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations

When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations

3 Jun 2021ICLR 2022 4arXiv:2106.01548archive 2025-07-28

Xiangning Chen, Cho-Jui Hsieh, Boqing Gong

Vision Transformers (ViTs) and MLPs signal further efforts on replacing hand-wired features or inductive biases with general-purpose neural architectures. Existing works empower the models by massive data, such as large-scale pre-training and/or repeated strong data augmentations, and still report optimization-related problems (e.g., sensitivity to initialization and learning rates). Hence, this paper investigates ViTs and MLP-Mixers from the lens of loss geometry, intending to improve the models' data efficiency at training and generalization at inference. Visualization and Hessian reveal extremely sharp local minima of converged models. By promoting smoothness with a recently proposed sharpness-aware optimizer, we substantially improve the accuracy and robustness of ViTs and MLP-Mixers on various tasks spanning supervised, adversarial, contrastive, and transfer learning (e.g., +5.3% and +11.0% top-1 accuracy on ImageNet for ViT-B/16 and Mixer-B/16, respectively, with the simple Inception-style preprocessing). We show that the improved smoothness attributes to sparser active neurons in the first few layers. The resultant ViTs outperform ResNets of similar size and throughput when trained from scratch on ImageNet without large-scale pre-training or strong data augmentations. Model checkpoints are available at https://github.com/google-research/vision_transformer.

PaperPDFConference PDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

google-research/vision_transformer officialmentioned in papermentioned on GitHubjaxSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv:2010.11929; that paper's own run record: 307 ran (of which 165 constructed an object rather than computing a result; 286 with no instrument failure: 8 honoured, 2 violated, 276 with no contract checked; 21 where Syntology's instrument failed) · 112 unverified) and MLP-Mixer: An all-MLP Architecture for Vision (arXiv:2105.01601; that paper's own run record: 114 ran (of which 77 constructed an object rather than computing a result; 104 with no instrument failure: 2 honoured, 1 violated, 101 with no contract checked; 10 where Syntology's instrument failed) · 20 unverified); nothing checks that they implement this paper's method. report
ttt496/VisionTransformer mentioned on GitHubjaxSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv:2010.11929; that paper's own run record: 307 ran (of which 165 constructed an object rather than computing a result; 286 with no instrument failure: 8 honoured, 2 violated, 276 with no contract checked; 21 where Syntology's instrument failed) · 112 unverified) and MLP-Mixer: An all-MLP Architecture for Vision (arXiv:2105.01601; that paper's own run record: 114 ran (of which 77 constructed an object rather than computing a result; 104 with no instrument failure: 2 honoured, 1 violated, 101 with no contract checked; 10 where Syntology's instrument failed) · 20 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Domain GeneralizationFine-Grained Image ClassificationImage ClassificationTransfer Learning

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Domain Generalization ImageNet-C ViT-B/16-SAM Top 1 Accuracy 56.5 #44 of 47 Archive leaderboard report
Domain Generalization ImageNet-C ResNet-152x2-SAM Top 1 Accuracy 55 #45 of 47 Archive leaderboard report
Domain Generalization ImageNet-C Mixer-B/8-SAM Top 1 Accuracy 48.9 #47 of 47 Archive leaderboard report
Domain Generalization ImageNet-R ResNet-152x2-SAM Top-1 Error Rate 71.9 #37 of 39 Archive leaderboard report
Domain Generalization ImageNet-R ViT-B/16-SAM Top-1 Error Rate 73.6 #38 of 39 Archive leaderboard report
Domain Generalization ImageNet-R Mixer-B/8-SAM Top-1 Error Rate 76.5 #39 of 39 Archive leaderboard report
Fine-Grained Image Classification Oxford-IIIT Pets ResNet-152-SAM Accuracy 93.3 #10 of 19 Archive leaderboard report
Fine-Grained Image Classification Oxford-IIIT Pets ViT-B/16- SAM Accuracy 93.1 #11 of 19 Archive leaderboard report
Fine-Grained Image Classification Oxford-IIIT Pets ViT-S/16- SAM Accuracy 92.9 #12 of 19 Archive leaderboard report
Fine-Grained Image Classification Oxford-IIIT Pets Mixer-B/16- SAM Accuracy 92.5 #13 of 19 Archive leaderboard report
Fine-Grained Image Classification Oxford-IIIT Pets ResNet-50-SAM Accuracy 91.6 #14 of 19 Archive leaderboard report
Fine-Grained Image Classification Oxford-IIIT Pets Mixer-S/16- SAM Accuracy 88.7 #15 of 19 Archive leaderboard report
Image Classification CIFAR-10 ViT-B/16- SAM Percentage correct 98.6 #35 of 265 Archive leaderboard report
Image Classification CIFAR-10 ResNet-152-SAM Percentage correct 98.2 #51 of 265 Archive leaderboard report
Image Classification CIFAR-10 ViT-S/16- SAM Percentage correct 98.2 #52 of 265 Archive leaderboard report
Image Classification CIFAR-10 Mixer-B/16- SAM Percentage correct 97.8 #70 of 265 Archive leaderboard report
Image Classification CIFAR-10 ResNet-50-SAM Percentage correct 97.4 #86 of 265 Archive leaderboard report
Image Classification CIFAR-10 Mixer-S/16- SAM Percentage correct 96.1 #120 of 265 Archive leaderboard report
Image Classification CIFAR-100 ViT-B/16- SAM Percentage correct 89.1 #33 of 211 Archive leaderboard report
Image Classification CIFAR-100 ViT-S/16- SAM Percentage correct 87.6 #42 of 211 Archive leaderboard report
Image Classification CIFAR-100 Mixer-B/16- SAM Percentage correct 86.4 #53 of 211 Archive leaderboard report
Image Classification CIFAR-100 ResNet-50-SAM Percentage correct 85.2 #66 of 211 Archive leaderboard report
Image Classification CIFAR-100 Mixer-S/16- SAM Percentage correct 82.4 #105 of 211 Archive leaderboard report
Image Classification Flowers-102 ViT-B/16- SAM Accuracy 91.8 #44 of 52 Archive leaderboard report
Image Classification Flowers-102 ViT-S/16- SAM Accuracy 91.5 #45 of 52 Archive leaderboard report
Image Classification Flowers-102 ResNet-152-SAM Accuracy 91.1 #46 of 52 Archive leaderboard report
Image Classification Flowers-102 ResNet-50-SAM Accuracy 90 #47 of 52 Archive leaderboard report
Image Classification Flowers-102 Mixer-B/16- SAM Accuracy 90 #48 of 52 Archive leaderboard report
Image Classification Flowers-102 Mixer-S/16- SAM Accuracy 87.9 #50 of 52 Archive leaderboard report
Image Classification ImageNet ResNet-152x2-SAM Number of params 236M #666 of 1060 Archive leaderboard report
Image Classification ImageNet ResNet-152x2-SAM Top 1 Accuracy 81.1% #666 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-B/16-SAM Number of params 87M #731 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-B/16-SAM Top 1 Accuracy 79.9% #731 of 1060 Archive leaderboard report
Image Classification ImageNet Mixer-B/8-SAM Number of params 64M #795 of 1060 Archive leaderboard report
Image Classification ImageNet Mixer-B/8-SAM Top 1 Accuracy 79% #795 of 1060 Archive leaderboard report
Image Classification ImageNet ReaL ResNet-152x2-SAM Accuracy 86.4% #37 of 57 Archive leaderboard report
Image Classification ImageNet ReaL ViT-B/16-SAM Accuracy 85.2% #43 of 57 Archive leaderboard report
Image Classification ImageNet ReaL Mixer-B/8-SAM Accuracy 84.4% #45 of 57 Archive leaderboard report
Image Classification ImageNet V2 ResNet-152x2-SAM Top 1 Accuracy 69.6 #26 of 33 Archive leaderboard report
Image Classification ImageNet V2 ViT-B/16-SAM Top 1 Accuracy 67.5 #30 of 33 Archive leaderboard report
Image Classification ImageNet V2 Mixer-B/8-SAM Top 1 Accuracy 65.5 #32 of 33 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections