{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/coatnet-marrying-convolution-and-attention","title":"CoAtNet: Marrying Convolution and Attention for All Data Sizes","arxiv_id":"2106.04803","date":"2021-06-09","proceeding":"NeurIPS 2021 12","authors":["Zihang Dai","Hanxiao Liu","Quoc V. Le","Mingxing Tan"],"abstract":"Transformers have attracted increasing interests in computer vision, but they still fall behind state-of-the-art convolutional networks. In this work, we show that while Transformers tend to have larger model capacity, their generalization can be worse than convolutional networks due to the lack of the right inductive bias. To effectively combine the strengths from both architectures, we present CoAtNets(pronounced \"coat\" nets), a family of hybrid models built from two key insights: (1) depthwise Convolution and self-Attention can be naturally unified via simple relative attention; (2) vertically stacking convolution layers and attention layers in a principled way is surprisingly effective in improving generalization, capacity and efficiency. Experiments show that our CoAtNets achieve state-of-the-art performance under different resource constraints across various datasets: Without extra data, CoAtNet achieves 86.0% ImageNet top-1 accuracy; When pre-trained with 13M images from ImageNet-21K, our CoAtNet achieves 88.56% top-1 accuracy, matching ViT-huge pre-trained with 300M images from JFT-300M while using 23x less data; Notably, when we further scale up CoAtNet with JFT-3B, it achieves 90.88% top-1 accuracy on ImageNet, establishing a new state-of-the-art result.","url_abs":"https://arxiv.org/abs/2106.04803v2","url_pdf":"https://arxiv.org/pdf/2106.04803v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/Burf/CoAtNet-Tensorflow2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/LongLeCE/CoAtNet-PyTorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/chinhsuanwu/coatnet-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/nqt228/CoAtNet-tensorflow","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/pranavsinghps1/dedl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/rwightman/pytorch-image-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/tyeso/Image_Classification_with_CoAtNet_and_ResNet18","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/MS-Mind/MS-Code-02/tree/main/configs/coat","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/Mind23-2/MindCode-19","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/canturan10/satellighte","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/hw666666666666/CoAtNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/coatnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/mindspore-courses/External-Attention-MindSpore/blob/main/model/attention/CoAtNet.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"coatnet-marrying-convolution-and-attention","repo_url":"https://github.com/xmu-xiaoma666/External-Attention-pytorch/blob/master/model/attention/CoAtNet.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"depthwise-convolution","method_name":"Depthwise Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-gashissdb","task":"Image Classification","dataset":"GasHisSDB","model":"CoAtNet-1","rank_in_archive_order":1,"of":8,"metrics":{"Accuracy":"98.74","F1-Score":"99.38","Precision":"99.97"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CoAtNet-3 @384","rank_in_archive_order":40,"of":1060,"metrics":{"GFLOPs":"114","Top 1 Accuracy":"88.52%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CoAtNet-3 (21k)","rank_in_archive_order":78,"of":1060,"metrics":{"Top 1 Accuracy":"87.6%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CoAtNet-2 (21k)","rank_in_archive_order":103,"of":1060,"metrics":{"Top 1 Accuracy":"87.1%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CoAtNet-3","rank_in_archive_order":314,"of":1060,"metrics":{"GFLOPs":"34.7","Number of params":"168M","Top 1 Accuracy":"84.5%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CoAtNet-2","rank_in_archive_order":358,"of":1060,"metrics":{"GFLOPs":"15.7","Number of params":"75M","Top 1 Accuracy":"84.1%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CoAtNet-1","rank_in_archive_order":443,"of":1060,"metrics":{"GFLOPs":"8.4","Number of params":"42M","Top 1 Accuracy":"83.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CoAtNet-0","rank_in_archive_order":627,"of":1060,"metrics":{"GFLOPs":"4.2","Number of params":"25M","Top 1 Accuracy":"81.6%"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2106.04803","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.04803"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rwightman/pytorch-image-models","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MS-Mind/MS-Code-02/tree/main/configs/coat","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chinhsuanwu/coatnet-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tyeso/Image_Classification_with_CoAtNet_and_ResNet18","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xmu-xiaoma666/External-Attention-pytorch/blob/master/model/attention/CoAtNet.py","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/coatnet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hw666666666666/CoAtNet","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/LongLeCE/CoAtNet-PyTorch","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Burf/CoAtNet-Tensorflow2","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/canturan10/satellighte","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/nqt228/CoAtNet-tensorflow","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-courses/External-Attention-MindSpore/blob/main/model/attention/CoAtNet.py","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pranavsinghps1/dedl","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Mind23-2/MindCode-19","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran_honours":1,"unverified":3},"by_repo_kind":{"listed":{"samples":4,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"c825b672e6f0b3bc","entry":"conv_3x3_bn","repo":"chinhsuanwu/coatnet-pytorch","repo_kind":"listed","path":"coatnet.py","file_url":"https://github.com/chinhsuanwu/coatnet-pytorch/blob/HEAD/coatnet.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c825b672e6f0b3bc"}},{"code_sha256_prefix":"f6b944f50d3f15ae","entry":"count_parameters","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f6b944f50d3f15ae"}},{"code_sha256_prefix":"8aeee8782df9ac39","entry":"coatnet","repo":"Burf/CoAtNet-Tensorflow2","repo_kind":"listed","path":"coatnet/coatnet.py","file_url":"https://github.com/Burf/CoAtNet-Tensorflow2/blob/HEAD/coatnet/coatnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8aeee8782df9ac39"}},{"code_sha256_prefix":"9dbc2c10a3104a03","entry":"coatnet0","repo":"Burf/CoAtNet-Tensorflow2","repo_kind":"listed","path":"coatnet/coatnet.py","file_url":"https://github.com/Burf/CoAtNet-Tensorflow2/blob/HEAD/coatnet/coatnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9dbc2c10a3104a03"}},{"code_sha256_prefix":"db9a51f9739f40ea","entry":"coatnet1","repo":"Burf/CoAtNet-Tensorflow2","repo_kind":"listed","path":"coatnet/coatnet.py","file_url":"https://github.com/Burf/CoAtNet-Tensorflow2/blob/HEAD/coatnet/coatnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"db9a51f9739f40ea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}