{"url":"/method/cvt","slug":"cvt","name":"CvT","full_name":"Convolutional Vision Transformer","full_name_withheld":false,"description_markdown":"The **Convolutional vision Transformer (CvT)** is an architecture which incorporates convolutions into the [Transformer](https://paperswithcode.com/method/transformer). The CvT design introduces convolutions to two core sections of the ViT architecture.\r\n\r\nFirst, the Transformers are partitioned into multiple stages that form a hierarchical structure of Transformers. The beginning of each stage consists of a convolutional token embedding that performs an overlapping [convolution](https://paperswithcode.com/method/convolution) operation with stride on a 2D-reshaped token map (i.e., reshaping flattened token sequences back to the spatial grid), followed by [layer normalization](https://paperswithcode.com/method/layer-normalization). This allows the model to not only capture local information, but also progressively decrease the sequence length while simultaneously increasing the dimension of token features across stages, achieving spatial downsampling while concurrently increasing the number of feature maps, as is performed in CNNs. \r\n\r\nSecond, the linear projection prior to every self-attention block in the Transformer module is replaced with a proposed convolutional projection, which employs a s × s depth-wise separable convolution operation on an 2D-reshaped token map. This allows the model to further capture local spatial context and reduce semantic ambiguity in the attention mechanism. It also permits management of computational complexity, as the stride of convolution can be used to subsample the key and value matrices to improve efficiency by 4× or more, with minimal degradation of performance.","description_state":"present","introduced_year":null,"introduced_by":{"title":"CvT: Introducing Convolutions to Vision Transformers","paper":"/paper/cvt-introducing-convolutions-to-vision","first_author":"Haiping Wu","n_authors":7,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/cvt-introducing-convolutions-to-vision"},"source":{"url":"https://arxiv.org/abs/2103.15808v1","title":"CvT: Introducing Convolutions to Vision Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Computer Vision","area_id":"computer-vision","collection":"Image Models","url":"/methods/category/image-models","pwc_aliases":[]},{"area":"Computer Vision","area_id":"computer-vision","collection":"Vision Transformers","url":"/methods/category/vision-transformers","pwc_aliases":["vision-transformer"]}],"n_papers_tagged":12,"archive_num_papers":12,"papers_newest_first":[{"paper":"/paper/residual-transformer-fusion-network-for-salt-1","title":"Residual Transformer Fusion Network for Salt and Pepper Image Denoising","date":"2025-02-13","arxiv_id":"2502.09000","n_code_links":0,"syntology":null},{"paper":"/paper/spectral-convolutional-transformer","title":"Heracles: A Hybrid SSM-Transformer Model for High-Resolution Image and Time-Series Analysis","date":"2024-03-26","arxiv_id":"2403.18063","n_code_links":2,"syntology":{"ran":4,"of":4,"unverified":0,"pointer_only":4}},{"paper":null,"title":"A Comparison of Deep Learning Models for Proton Background Rejection with the AMS Electromagnetic Calorimeter","date":"2024-02-26","arxiv_id":"2402.16285","n_code_links":0,"syntology":null},{"paper":null,"title":"On Convolutional Vision Transformers for Yield Prediction","date":"2024-02-08","arxiv_id":"2402.05557","n_code_links":0,"syntology":null},{"paper":null,"title":"Making Vision Transformers Truly Shift-Equivariant","date":"2023-05-25","arxiv_id":"2305.16316","n_code_links":0,"syntology":null},{"paper":null,"title":"A Novel Framework for Decentralized Dynamic Resource Allocation Using Voronoi Tessellations","date":"2022-11-30","arxiv_id":"2212.00140","n_code_links":0,"syntology":null},{"paper":null,"title":"Online Continual Learning with Contrastive Vision Transformer","date":"2022-07-24","arxiv_id":"2207.13516","n_code_links":0,"syntology":null},{"paper":"/paper/sima-simple-softmax-free-attention-for-vision","title":"SimA: Simple Softmax-free Attention for Vision Transformers","date":"2022-06-17","arxiv_id":"2206.08898","n_code_links":1,"syntology":{"ran":0,"of":3,"unverified":3,"pointer_only":0}},{"paper":"/paper/convolutional-xformers-for-vision","title":"Convolutional Xformers for Vision","date":"2022-01-25","arxiv_id":"2201.10271","n_code_links":1,"syntology":null},{"paper":null,"title":"Energy-optimal Design and Control of Electric Powertrains under Motor Thermal Constraints","date":"2021-11-15","arxiv_id":"2111.07711","n_code_links":0,"syntology":null},{"paper":null,"title":"Convolutional Gated MLP: Combining Convolutions & gMLP","date":"2021-11-06","arxiv_id":"2111.03940","n_code_links":0,"syntology":null},{"paper":"/paper/cvt-introducing-convolutions-to-vision","title":"CvT: Introducing Convolutions to Vision Transformers","date":"2021-03-29","arxiv_id":"2103.15808","n_code_links":16,"syntology":{"ran":29,"of":47,"unverified":18,"pointer_only":4}}],"papers_shown":12,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":2},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":2},{"task":"/task/image-classification","name":"image-classification","papers":2},{"task":"/task/continual-learning","name":"Continual Learning","papers":1},{"task":"/task/contrastive-learning","name":"Contrastive Learning","papers":1},{"task":"/task/deep-learning","name":"Deep Learning","papers":1},{"task":"/task/denoising","name":"Denoising","papers":1},{"task":null,"name":"GPU","papers":1},{"task":"/task/image-denoising","name":"Image Denoising","papers":1},{"task":"/task/image-reconstruction","name":"Image Reconstruction","papers":1},{"task":"/task/inductive-bias","name":"Inductive Bias","papers":1},{"task":"/task/instance-segmentation","name":"Instance Segmentation","papers":1},{"task":"/task/mamba","name":"Mamba","papers":1},{"task":"/task/prediction","name":"Prediction","papers":1},{"task":"/task/state-space-models","name":"State Space Models","papers":1},{"task":"/task/time-series-1","name":"Time Series","papers":1},{"task":"/task/time-series","name":"Time Series Analysis","papers":1}],"tasks_shown":18,"n_tasks":18,"usage_by_year":[{"year":"2021","papers":3},{"year":"2022","papers":4},{"year":"2023","papers":1},{"year":"2024","papers":3},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/cvt"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}