Papers › Co-training 2ᴸ Submodels for Visual Recognition

Co-training 2ᴸ Submodels for Visual Recognition

9 Dec 2022arXiv:2212.04884archive 2025-07-28

Hugo Touvron, Matthieu Cord, Maxime Oquab, Piotr Bojanowski, Jakob Verbeek, Hervé Jégou

We introduce submodel co-training, a regularization method related to co-training, self-distillation and stochastic depth. Given a neural network to be trained, for each sample we implicitly instantiate two altered networks, ``submodels'', with stochastic depth: we activate only a subset of the layers. Each network serves as a soft teacher to the other, by providing a loss that complements the regular loss provided by the one-hot label. Our approach, dubbed cosub, uses a single set of weights, and does not involve a pre-trained external model or temporal averaging. Experimentally, we show that submodel co-training is effective to train backbones for recognition tasks such as image classification and semantic segmentation. Our approach is compatible with multiple architectures, including RegNet, ViT, PiT, XCiT, Swin and ConvNext. Our training strategy improves their results in comparable settings. For instance, a ViT-B pretrained with cosub on ImageNet-21k obtains 87.4% top-1 acc. @448 on ImageNet-val.

PaperPDFCode

In Syntology View this paper on Syntology, its page in Syntology's graph. That page lists the repositories linked to the paper, the abstract and the calls for agents.

Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

For agents, on Syntology's MCP service (how to connect):

Code

facebookresearch/deit mentioned in paperpytorchSyntology: no sample linked to this paper (harvested for another paper). Syntology's graph links these samples to Augmenting Convolutional networks with attention-based aggregation (arXiv:2112.13692; that paper's own run record: 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified) and Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It (arXiv:2403.14715; that paper's own run record: 9 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified); nothing checks that they implement this paper's method. report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

A paper named beside a repository with no sample linked to this paper is shown with that paper's own run record, not this paper's: “ran” means executed on a synthesized input, not that the code is correct, and “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. Papers are listed in arXiv-id order, at most three per repository.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Image ClassificationSemantic Segmentationimage-classification

Results from the paper archive 2025-07-28

TaskDatasetModelMetricValueRank at snapshotLeaderboardReport
Image Classification ImageNet ViT-H@224 (cosub) Top 1 Accuracy 88.0% #62 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-L@224 (cosub) Top 1 Accuracy 87.5% #82 of 1060 Archive leaderboard report
Image Classification ImageNet Swin-L@224 (cosub) Top 1 Accuracy 87.1% #102 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-B@224 (cosub) Top 1 Accuracy 86.3% #155 of 1060 Archive leaderboard report
Image Classification ImageNet Swin-B@224 (cosub) Top 1 Accuracy 86.2% #165 of 1060 Archive leaderboard report
Image Classification ImageNet ConvNeXt-B@224 (cosub) Top 1 Accuracy 85.8% #189 of 1060 Archive leaderboard report
Image Classification ImageNet PiT-B@224 (cosub) Top 1 Accuracy 85.8% #190 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-M@224 (cosub) Top 1 Accuracy 85.0% #263 of 1060 Archive leaderboard report
Image Classification ImageNet RegnetY16GF@224 (cosub) Top 1 Accuracy 84.2% #336 of 1060 Archive leaderboard report
Image Classification ImageNet ViT-S@224 (cosub) Top 1 Accuracy 83.1% #466 of 1060 Archive leaderboard report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections