Papers › DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets
DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets
Harsh Rangwani, Pradipto Mondal, Mayank Mishra, Ashish Ramayee Asokan, R. Venkatesh Babu
Vision Transformer (ViT) has emerged as a prominent architecture for various computer vision tasks. In ViT, we divide the input image into patch tokens and process them through a stack of self attention blocks. However, unlike Convolutional Neural Networks (CNN), ViTs simple architecture has no informative inductive bias (e.g., locality,etc. ). Due to this, ViT requires a large amount of data for pre-training. Various data efficient approaches (DeiT) have been proposed to train ViT on balanced datasets effectively. However, limited literature discusses the use of ViT for datasets with long-tailed imbalances. In this work, we introduce DeiT-LT to tackle the problem of training ViTs from scratch on long-tailed datasets. In DeiT-LT, we introduce an efficient and effective way of distillation from CNN via distillation DIST token by using out-of-distribution images and re-weighting the distillation loss to enhance focus on tail classes. This leads to the learning of local CNN-like features in early ViT blocks, improving generalization for tail classes. Further, to mitigate overfitting, we propose distilling from a flat CNN teacher, which leads to learning low-rank generalizable features for DIST tokens across all ViT blocks. With the proposed DeiT-LT scheme, the distillation DIST token becomes an expert on the tail classes, and the classifier CLS token becomes an expert on the head classes. The experts help to effectively learn features corresponding to both the majority and minority classes using a distinct set of tokens within the same ViT architecture. We show the effectiveness of DeiT-LT for training ViT from scratch on datasets ranging from small-scale CIFAR-10 LT to large-scale iNaturalist-2018.
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Image Classification | iNaturalist | b_22DeiT-LT(ours) | Overall | 75.1 | #19 of 19 | Archive leaderboard | report |
| Long-tail Learning | CIFAR-10-LT (ρ=100) | DeiT-LT | Error Rate | 12.5 | #5 of 28 | Archive leaderboard | report |
| Long-tail Learning | CIFAR-10-LT (ρ=50) | DeiT-LT | Error Rate | 10.2 | #3 of 8 | Archive leaderboard | report |
| Long-tail Learning | CIFAR-100-LT (ρ=100) | DeiT-LT | Error Rate | 44.4 | #11 of 66 | Archive leaderboard | report |
| Long-tail Learning | CIFAR-100-LT (ρ=50) | DeiT-LT | Error Rate | 39.5 | #8 of 25 | Archive leaderboard | report |
| Long-tail Learning | ImageNet-LT | DeiT-LT | Top-1 Accuracy | 59.1 | #16 of 69 | Archive leaderboard | report |
| Long-tail Learning | iNaturalist 2018 | DeiT-LT | Top-1 Accuracy | 75.1% | #15 of 43 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections