{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/incorporating-convolution-designs-into-visual","title":"Incorporating Convolution Designs into Visual Transformers","arxiv_id":"2103.11816","date":"2021-03-22","proceeding":"ICCV 2021 10","authors":["Kun Yuan","Shaopeng Guo","Ziwei Liu","Aojun Zhou","Fengwei Yu","Wei Wu"],"abstract":"Motivated by the success of Transformers in natural language processing (NLP) tasks, there emerge some attempts (e.g., ViT and DeiT) to apply Transformers to the vision domain. However, pure Transformer architectures often require a large amount of training data or extra supervision to obtain comparable performance with convolutional neural networks (CNNs). To overcome these limitations, we analyze the potential drawbacks when directly borrowing Transformer architectures from NLP. Then we propose a new \\textbf{Convolution-enhanced image Transformer (CeiT)} which combines the advantages of CNNs in extracting low-level features, strengthening locality, and the advantages of Transformers in establishing long-range dependencies. Three modifications are made to the original Transformer: \\textbf{1)} instead of the straightforward tokenization from raw input images, we design an \\textbf{Image-to-Tokens (I2T)} module that extracts patches from generated low-level features; \\textbf{2)} the feed-froward network in each encoder block is replaced with a \\textbf{Locally-enhanced Feed-Forward (LeFF)} layer that promotes the correlation among neighboring tokens in the spatial dimension; \\textbf{3)} a \\textbf{Layer-wise Class token Attention (LCA)} is attached at the top of the Transformer that utilizes the multi-level representations. Experimental results on ImageNet and seven downstream tasks show the effectiveness and generalization ability of CeiT compared with previous Transformers and state-of-the-art CNNs, without requiring a large amount of training data and extra CNN teachers. Besides, CeiT models also demonstrate better convergence with $3\\times$ fewer training iterations, which can reduce the training cost significantly\\footnote{Code and models will be released upon acceptance.}.","url_abs":"https://arxiv.org/abs/2103.11816v2","url_pdf":"https://arxiv.org/pdf/2103.11816v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"incorporating-convolution-designs-into-visual","repo_url":"https://github.com/coeusguo/ceit","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"incorporating-convolution-designs-into-visual","repo_url":"https://github.com/rishikksh20/CeiT-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"incorporating-convolution-designs-into-visual","repo_url":"https://github.com/mindspore-courses/External-Attention-MindSpore/blob/main/model/backbone/CeiT.py","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"ceit","method_name":"CeiT"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"CeiT-S (384 finetune resolution)","rank_in_archive_order":16,"of":265,"metrics":{"Percentage correct":"99.1"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"CeiT-S","rank_in_archive_order":23,"of":265,"metrics":{"Percentage correct":"99"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"CeiT-T","rank_in_archive_order":40,"of":265,"metrics":{"Percentage correct":"98.5"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"CeiT-S","rank_in_archive_order":16,"of":211,"metrics":{"Percentage correct":"91.8"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"CeiT-S (384 finetune resolution)","rank_in_archive_order":17,"of":211,"metrics":{"Percentage correct":"91.8"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"CeiT-T","rank_in_archive_order":31,"of":211,"metrics":{"Percentage correct":"89.4"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"CeiT-T (384 finetune resolution)","rank_in_archive_order":40,"of":211,"metrics":{"Percentage correct":"88"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-flowers-102","task":"Image Classification","dataset":"Flowers-102","model":"CeiT-S (384 finetune resolution)","rank_in_archive_order":22,"of":52,"metrics":{"Accuracy":"98.6"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-flowers-102","task":"Image Classification","dataset":"Flowers-102","model":"CeiT-S","rank_in_archive_order":28,"of":52,"metrics":{"Accuracy":"98.2"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-flowers-102","task":"Image Classification","dataset":"Flowers-102","model":"CeiT-T (384 finetune resolution)","rank_in_archive_order":35,"of":52,"metrics":{"Accuracy":"97.8"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-flowers-102","task":"Image Classification","dataset":"Flowers-102","model":"CeiT-T","rank_in_archive_order":40,"of":52,"metrics":{"Accuracy":"96.9"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CeiT-S (384 finetune res)","rank_in_archive_order":437,"of":1060,"metrics":{"GFLOPs":"12.9","Number of params":"24.2M","Top 1 Accuracy":"83.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CeiT-S","rank_in_archive_order":579,"of":1060,"metrics":{"GFLOPs":"4.5","Top 1 Accuracy":"82%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CeiT-T (384 finetune res)","rank_in_archive_order":801,"of":1060,"metrics":{"GFLOPs":"3.6","Top 1 Accuracy":"78.8%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"CeiT-T","rank_in_archive_order":914,"of":1060,"metrics":{"GFLOPs":"1.2","Number of params":"6.4M","Top 1 Accuracy":"76.4%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet-real","task":"Image Classification","dataset":"ImageNet ReaL","model":"CeiT-S (384 finetune res)","rank_in_archive_order":28,"of":57,"metrics":{"Accuracy":"88.1%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet-real","task":"Image Classification","dataset":"ImageNet ReaL","model":"CeiT-S","rank_in_archive_order":34,"of":57,"metrics":{"Accuracy":"87.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet-real","task":"Image Classification","dataset":"ImageNet ReaL","model":"CeiT-T","rank_in_archive_order":47,"of":57,"metrics":{"Accuracy":"83.6%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-oxford-iiit-pets-1","task":"Image Classification","dataset":"Oxford-IIIT Pets","model":"CeiT-S (384 finetune resolution)","rank_in_archive_order":1,"of":6,"metrics":{"Accuracy":"94.9"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-oxford-iiit-pets-1","task":"Image Classification","dataset":"Oxford-IIIT Pets","model":"CeiT-S","rank_in_archive_order":3,"of":6,"metrics":{"Accuracy":"94.6"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-oxford-iiit-pets-1","task":"Image Classification","dataset":"Oxford-IIIT Pets","model":"CeiT-T (384 finetune resolution)","rank_in_archive_order":4,"of":6,"metrics":{"Accuracy":"94.5"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-oxford-iiit-pets-1","task":"Image Classification","dataset":"Oxford-IIIT Pets","model":"CeiT-T","rank_in_archive_order":5,"of":6,"metrics":{"Accuracy":"93.8"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-stanford-cars","task":"Image Classification","dataset":"Stanford Cars","model":"CeiT-S (384 finetune resolution)","rank_in_archive_order":7,"of":24,"metrics":{"Accuracy":"94.1"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-stanford-cars","task":"Image Classification","dataset":"Stanford Cars","model":"CeiT-S","rank_in_archive_order":10,"of":24,"metrics":{"Accuracy":"93.2"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-stanford-cars","task":"Image Classification","dataset":"Stanford Cars","model":"CeiT-T (384 finetune resolution)","rank_in_archive_order":12,"of":24,"metrics":{"Accuracy":"93"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-stanford-cars","task":"Image Classification","dataset":"Stanford Cars","model":"CeiT-T","rank_in_archive_order":14,"of":24,"metrics":{"Accuracy":"90.5"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-inaturalist-2018","task":"Image Classification","dataset":"iNaturalist 2018","model":"CeiT-S (384 finetune resolution)","rank_in_archive_order":19,"of":60,"metrics":{"Top-1 Accuracy":"79.4%"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-inaturalist-2018","task":"Image Classification","dataset":"iNaturalist 2018","model":"CeiT-S","rank_in_archive_order":32,"of":60,"metrics":{"Top-1 Accuracy":"73.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-inaturalist-2018","task":"Image Classification","dataset":"iNaturalist 2018","model":"CeiT-T (384 finetune resolution)","rank_in_archive_order":34,"of":60,"metrics":{"Top-1 Accuracy":"72.2%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-inaturalist-2018","task":"Image Classification","dataset":"iNaturalist 2018","model":"CeiT-T","rank_in_archive_order":50,"of":60,"metrics":{"Top-1 Accuracy":"64.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-inaturalist-2019","task":"Image Classification","dataset":"iNaturalist 2019","model":"CeiT-S (384 finetune resolution)","rank_in_archive_order":9,"of":22,"metrics":{"Top-1 Accuracy":"82.7"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-inaturalist-2019","task":"Image Classification","dataset":"iNaturalist 2019","model":"CeiT-S","rank_in_archive_order":12,"of":22,"metrics":{"Top-1 Accuracy":"78.9"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-inaturalist-2019","task":"Image Classification","dataset":"iNaturalist 2019","model":"CeiT-T (384 finetune resolution)","rank_in_archive_order":13,"of":22,"metrics":{"Top-1 Accuracy":"77.9"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-inaturalist-2019","task":"Image Classification","dataset":"iNaturalist 2019","model":"CeiT-T","rank_in_archive_order":16,"of":22,"metrics":{"Top-1 Accuracy":"72.8"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2103.11816","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2103.11816"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mindspore-courses/External-Attention-MindSpore/blob/main/model/backbone/CeiT.py","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rishikksh20/CeiT-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/coeusguo/ceit","reach":null}],"summary":{"ran":9,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1},"listed":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"88ac331124a64a60","entry":"Attention","repo":"coeusguo/ceit","repo_kind":"official","path":"ceit_model.py","file_url":"https://github.com/coeusguo/ceit/blob/HEAD/ceit_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88ac331124a64a60"}},{"code_sha256_prefix":"ef3f1eb732093bcb","entry":"AttentionLCA","repo":"coeusguo/ceit","repo_kind":"official","path":"ceit_model.py","file_url":"https://github.com/coeusguo/ceit/blob/HEAD/ceit_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ef3f1eb732093bcb"}},{"code_sha256_prefix":"e5a66231dac09f8c","entry":"Block","repo":"coeusguo/ceit","repo_kind":"official","path":"ceit_model.py","file_url":"https://github.com/coeusguo/ceit/blob/HEAD/ceit_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e5a66231dac09f8c"}},{"code_sha256_prefix":"531854dd3e8dc7b0","entry":"HybridEmbed","repo":"coeusguo/ceit","repo_kind":"official","path":"ceit_model.py","file_url":"https://github.com/coeusguo/ceit/blob/HEAD/ceit_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"531854dd3e8dc7b0"}},{"code_sha256_prefix":"f7134b0e48a1485d","entry":"LCA","repo":"rishikksh20/CeiT-pytorch","repo_kind":"listed","path":"ceit.py","file_url":"https://github.com/rishikksh20/CeiT-pytorch/blob/HEAD/ceit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7134b0e48a1485d"}},{"code_sha256_prefix":"218a36c2869e1755","entry":"LCAttention","repo":"rishikksh20/CeiT-pytorch","repo_kind":"listed","path":"ceit.py","file_url":"https://github.com/rishikksh20/CeiT-pytorch/blob/HEAD/ceit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"218a36c2869e1755"}},{"code_sha256_prefix":"35eefd450a0413e9","entry":"LeFF","repo":"rishikksh20/CeiT-pytorch","repo_kind":"listed","path":"ceit.py","file_url":"https://github.com/rishikksh20/CeiT-pytorch/blob/HEAD/ceit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35eefd450a0413e9"}},{"code_sha256_prefix":"ceb88739eac75055","entry":"LocallyEnhancedFeedForward","repo":"coeusguo/ceit","repo_kind":"official","path":"ceit_model.py","file_url":"https://github.com/coeusguo/ceit/blob/HEAD/ceit_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ceb88739eac75055"}},{"code_sha256_prefix":"bbca7f007eecf184","entry":"TransformerLeFF","repo":"rishikksh20/CeiT-pytorch","repo_kind":"listed","path":"ceit.py","file_url":"https://github.com/rishikksh20/CeiT-pytorch/blob/HEAD/ceit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bbca7f007eecf184"}},{"code_sha256_prefix":"f706125b2848bd36","entry":"CeIT","repo":"coeusguo/ceit","repo_kind":"official","path":"ceit_model.py","file_url":"https://github.com/coeusguo/ceit/blob/HEAD/ceit_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f706125b2848bd36"}},{"code_sha256_prefix":"779c5d5035c71712","entry":"CeiT","repo":"rishikksh20/CeiT-pytorch","repo_kind":"listed","path":"ceit.py","file_url":"https://github.com/rishikksh20/CeiT-pytorch/blob/HEAD/ceit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"779c5d5035c71712"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}