</> Code with Papers by Syntology

Papers › Co-training 2ᴸ Submodels for Visual Recognition

Co-training 2ᴸ Submodels for Visual Recognition

9 Dec 2022arXiv:2212.04884archive 2025-07-28

Hugo Touvron, Matthieu Cord, Maxime Oquab, Piotr Bojanowski, Jakob Verbeek, Hervé Jégou

We introduce submodel co-training, a regularization method related to co-training, self-distillation and stochastic depth. Given a neural network to be trained, for each sample we implicitly instantiate two altered networks, ``submodels'', with stochastic depth: we activate only a subset of the layers. Each network serves as a soft teacher to the other, by providing a loss that complements the regular loss provided by the one-hot label. Our approach, dubbed cosub, uses a single set of weights, and does not involve a pre-trained external model or temporal averaging. Experimentally, we show that submodel co-training is effective to train backbones for recognition tasks such as image classification and semantic segmentation. Our approach is compatible with multiple architectures, including RegNet, ViT, PiT, XCiT, Swin and ConvNext. Our training strategy improves their results in comparable settings. For instance, a ViT-B pretrained with cosub on ImageNet-21k obtains 87.4% top-1 acc. @448 on ImageNet-val.

PaperPDFCode

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

facebookresearch/deit mentioned in paperpytorch report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image ClassificationSemantic Segmentationimage-classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Image Classification ImageNet ViT-H@224 (cosub) Top 1 Accuracy 88.0% #62 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-L@224 (cosub) Top 1 Accuracy 87.5% #82 of 1060 Archive leaderboard report
Image Classification ImageNet Swin-L@224 (cosub) Top 1 Accuracy 87.1% #102 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-B@224 (cosub) Top 1 Accuracy 86.3% #155 of 1060 Archive leaderboard report
Image Classification ImageNet Swin-B@224 (cosub) Top 1 Accuracy 86.2% #165 of 1060 Archive leaderboard report
Image Classification ImageNet ConvNeXt-B@224 (cosub) Top 1 Accuracy 85.8% #189 of 1060 Archive leaderboard report
Image Classification ImageNet PiT-B@224 (cosub) Top 1 Accuracy 85.8% #190 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-M@224 (cosub) Top 1 Accuracy 85.0% #263 of 1060 Archive leaderboard report
Image Classification ImageNet RegnetY16GF@224 (cosub) Top 1 Accuracy 84.2% #336 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-S@224 (cosub) Top 1 Accuracy 83.1% #466 of 1060 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections