{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/efficient-visual-pretraining-with-contrastive","title":"Efficient Visual Pretraining with Contrastive Detection","arxiv_id":"2103.10957","date":"2021-03-19","proceeding":"ICCV 2021 10","authors":["Olivier J. Hénaff","Skanda Koppula","Jean-Baptiste Alayrac","Aaron van den Oord","Oriol Vinyals","João Carreira"],"abstract":"Self-supervised pretraining has been shown to yield powerful representations for transfer learning. These performance gains come at a large computational cost however, with state-of-the-art methods requiring an order of magnitude more computation than supervised pretraining. We tackle this computational bottleneck by introducing a new self-supervised objective, contrastive detection, which tasks representations with identifying object-level features across augmentations. This objective extracts a rich learning signal per image, leading to state-of-the-art transfer accuracy on a variety of downstream tasks, while requiring up to 10x less pretraining. In particular, our strongest ImageNet-pretrained model performs on par with SEER, one of the largest self-supervised systems to date, which uses 1000x more pretraining data. Finally, our objective seamlessly handles pretraining on more complex images such as those in COCO, closing the gap with supervised transfer learning from COCO to PASCAL.","url_abs":"https://arxiv.org/abs/2103.10957v2","url_pdf":"https://arxiv.org/pdf/2103.10957v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"efficient-visual-pretraining-with-contrastive","repo_url":"https://github.com/deepmind/detcon","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"unanswered"}},{"paper_slug":"efficient-visual-pretraining-with-contrastive","repo_url":"https://github.com/isaaccorley/detcon-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"grouped-convolution","method_name":"Grouped Convolution"},{"method_slug":"lars","method_name":"LARS"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"regnety","method_name":"RegNetY"},{"method_slug":"seer","method_name":"SEER"},{"method_slug":"sigmoid-activation","method_name":"Sigmoid Activation"},{"method_slug":"squeeze-and-excitation-block","method_name":"Squeeze-and-Excitation Block"},{"method_slug":"swav","method_name":"SwAV"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/semantic-segmentation-on-cityscapes-val","task":"Semantic Segmentation","dataset":"Cityscapes val","model":"DetCon_B","rank_in_archive_order":65,"of":99,"metrics":{"mIoU":"77.0%"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.10957","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}