Papers › Masking meets Supervision: A Strong Learning Alliance
Masking meets Supervision: A Strong Learning Alliance
Byeongho Heo, Taekyung Kim, Sangdoo Yun, Dongyoon Han
Pre-training with random masked inputs has emerged as a novel trend in self-supervised training. However, supervised learning still faces a challenge in adopting masking augmentations, primarily due to unstable training. In this paper, we propose a novel way to involve masking augmentations dubbed Masked Sub-branch (MaskSub). MaskSub consists of the main-branch and sub-branch, the latter being a part of the former. The main-branch undergoes conventional training recipes, while the sub-branch merits intensive masking augmentations, during training. MaskSub tackles the challenge by mitigating adverse effects through a relaxed loss function similar to a self-distillation loss. Our analysis shows that MaskSub improves performance, with the training loss converging faster than in standard training, which suggests our method stabilizes the training process. We further validate MaskSub across diverse training scenarios and models, including DeiT-III training, MAE finetuning, CLIP finetuning, BERT training, and hierarchical architectures (ResNet and Swin Transformer). Our results show that MaskSub consistently achieves impressive performance gains across all the cases. MaskSub provides a practical and effective solution for introducing additional regularization under various training recipes. Code available at https://github.com/naver-ai/augsub
In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.
Code
Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.
Code Syntology ran Syntology
Not run by Syntology. Nothing on this page verifies that the listed code works.
Tasks
Results from the paper archive 2025-07-28
| Task | Dataset | Model | Metric | Value | Rank at snapshot | Leaderboard | Report |
|---|---|---|---|---|---|---|---|
| Image Classification | ImageNet | ViT-H @224 (DeiT-III + AugSub) | Number of params | 632M | #212 of 1060 | Archive leaderboard | report |
| Image Classification | ImageNet | ViT-H @224 (DeiT-III + AugSub) | Top 1 Accuracy | 85.7% | #212 of 1060 | Archive leaderboard | report |
| Image Classification | ImageNet | ViT-L @224 (DeiT-III + AugSub) | Number of params | 304M | #242 of 1060 | Archive leaderboard | report |
| Image Classification | ImageNet | ViT-L @224 (DeiT-III + AugSub) | Top 1 Accuracy | 85.3% | #242 of 1060 | Archive leaderboard | report |
| Image Classification | ImageNet | ViT-B @224 (DeiT-III + AugSub) | Number of params | 86.6M | #342 of 1060 | Archive leaderboard | report |
| Image Classification | ImageNet | ViT-B @224 (DeiT-III + AugSub) | Top 1 Accuracy | 84.2% | #342 of 1060 | Archive leaderboard | report |
| Self-Supervised Image Classification | ImageNet (finetuned) | MAE + AugSub finetune (ViT-H/14) | Number of Params | 632M | #9 of 65 | Archive leaderboard | report |
| Self-Supervised Image Classification | ImageNet (finetuned) | MAE + AugSub finetune (ViT-H/14) | Top 1 Accuracy | 87.2% | #9 of 65 | Archive leaderboard | report |
| Self-Supervised Image Classification | ImageNet (finetuned) | MAE + AugSub finetune (ViT-L/16) | Number of Params | 304M | #17 of 65 | Archive leaderboard | report |
| Self-Supervised Image Classification | ImageNet (finetuned) | MAE + AugSub finetune (ViT-L/16) | Top 1 Accuracy | 86.1% | #17 of 65 | Archive leaderboard | report |
| Self-Supervised Image Classification | ImageNet (finetuned) | MAE + AugSub finetune (ViT-B/16) | Number of Params | 87M | #41 of 65 | Archive leaderboard | report |
| Self-Supervised Image Classification | ImageNet (finetuned) | MAE + AugSub finetune (ViT-B/16) | Top 1 Accuracy | 83.9% | #41 of 65 | Archive leaderboard | report |
Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.
Methods
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections