Papers › Pre-training of Lightweight Vision Transformers on Small Datasets with Minimally Scaled Images
Pre-training of Lightweight Vision Transformers on Small Datasets with Minimally Scaled Images
Jen Hong Tan
Can a lightweight Vision Transformer (ViT) match or exceed the performance of Convolutional Neural Networks (CNNs) like ResNet on small datasets with small image resolutions? This report demonstrates that a pure ViT can indeed achieve superior performance through pre-training, using a masked auto-encoder technique with minimal image scaling. Our experiments on the CIFAR-10 and CIFAR-100 datasets involved ViT models with fewer than 3.65 million parameters and a multiply-accumulate (MAC) count below 0.27G, qualifying them as 'lightweight' models. Unlike previous approaches, our method attains state-of-the-art performance among similar lightweight transformer-based architectures without significantly scaling up images from CIFAR-10 and CIFAR-100. This achievement underscores the efficiency of our model, not only in handling small datasets but also in effectively processing images close to their original scale.
Code
No code repository is listed for this paper in the archive or in Syntology's graph.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Image Classification | CIFAR-10 | ViT (lightweight, MAE pretrained) | Percentage correct | 96.41 | #112 of 265 | Archive leaderboard | report |
| Image Classification | CIFAR-100 | ViT (lightweight, MAE pre-trained) | PARAMS | 3.64M | #139 of 211 | Archive leaderboard | report |
| Image Classification | CIFAR-100 | ViT (lightweight, MAE pre-trained) | Percentage correct | 78.27 | #139 of 211 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections