Papers › Discrete Representations Strengthen Vision Transformer Robustness
Discrete Representations Strengthen Vision Transformer Robustness
Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, Irfan Essa
Vision Transformer (ViT) is emerging as the state-of-the-art architecture for image recognition. While recent studies suggest that ViTs are more robust than their convolutional counterparts, our experiments find that ViTs trained on ImageNet are overly reliant on local textures and fail to make adequate use of shape information. ViTs thus have difficulties generalizing to out-of-distribution, real-world data. To address this deficiency, we present a simple and effective architecture modification to ViT's input layer by adding discrete tokens produced by a vector-quantized encoder. Different from the standard continuous pixel tokens, discrete tokens are invariant under small perturbations and contain less information individually, which promote ViTs to learn global information that is invariant. Experimental results demonstrate that adding discrete representation on four architecture variants strengthens ViT robustness by up to 12% across seven ImageNet robustness benchmarks while maintaining the performance on ImageNet.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Domain Generalization | ImageNet-C | DiscreteViT (Im21k) | Number of params | 87M | #11 of 47 | Archive leaderboard | report |
| Domain Generalization | ImageNet-C | DiscreteViT (Im21k) | mean Corruption Error (mCE) | 38.74 | #11 of 47 | Archive leaderboard | report |
| Domain Generalization | ImageNet-C | DrViT | mean Corruption Error (mCE) | 46.22 | #21 of 47 | Archive leaderboard | report |
| Domain Generalization | ImageNet-C | DiscreteViT | Number of params | 87M | #22 of 47 | Archive leaderboard | report |
| Domain Generalization | ImageNet-C | DiscreteViT | mean Corruption Error (mCE) | 46.22 | #22 of 47 | Archive leaderboard | report |
| Domain Generalization | ImageNet-R | DiscreteViT | Top-1 Error Rate | 44.74 | #20 of 39 | Archive leaderboard | report |
| Domain Generalization | ImageNet-Sketch | DrViT | Top-1 accuracy | 44.72 | #16 of 20 | Archive leaderboard | report |
| Domain Generalization | Stylized-ImageNet | DiscreteViT | Top 1 Accuracy | 22.19 | #3 of 3 | Archive leaderboard | report |
| Image Classification | ImageNet | DiscreteViT | Top 1 Accuracy | 85.07% | #262 of 1060 | Archive leaderboard | report |
| Image Classification | ObjectNet | ViT-B (Discrete 512x512) | Top-1 Accuracy | 46.62 | #35 of 106 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections