{"url":"/method/attention-augmented-convolution","slug":"attention-augmented-convolution","name":"Attention-augmented Convolution","full_name":"Attention-augmented Convolution","full_name_withheld":false,"description_markdown":"**Attention-augmented Convolution** is a type of [convolution](https://paperswithcode.com/method/convolution) with a two-dimensional relative self-attention mechanism that can replace convolutions as a stand-alone computational primitive for image classification. It employs [scaled-dot product attention](https://paperswithcode.com/method/scaled) and [multi-head attention](https://paperswithcode.com/method/multi-head-attention) as with [Transformers](https://paperswithcode.com/method/transformer).\r\n\r\nIt works by concatenating convolutional and attentional feature map. To see this, consider an original convolution operator with kernel size $k$, $F\\_{in}$ input filters and $F\\_{out}$ output filters. The corresponding attention augmented convolution can be written as\"\r\n\r\n$$\\text{AAConv}\\left(X\\right) = \\text{Concat}\\left[\\text{Conv}(X), \\text{MHA}(X)\\right] $$\r\n\r\n$X$ originates from an input tensor of shape $\\left(H, W, F\\_{in}\\right)$. This is flattened to become $X \\in \\mathbb{R}^{HW \\times F\\_{in}}$ which is passed into a multi-head attention module, as well as a convolution (see above).\r\n\r\nSimilarly to the convolution, the attention augmented convolution 1) is equivariant to translation and 2) can readily operate on inputs of different spatial dimensions.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Attention Augmented Convolutional Networks","paper":"/paper/190409925","first_author":"Irwan Bello","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/190409925"},"source":{"url":"https://arxiv.org/abs/1904.09925v5","title":"Attention Augmented Convolutional Networks","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/leaderj1001/Attention-Augmented-Conv2d/blob/1ce94a3072c2d9aabe258313b3a17c974d987411/AA-Wide-ResNet/attention_augmented_conv.py#L9","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Attention Modules","url":"/methods/category/attention-modules","pwc_aliases":[]},{"area":"Computer Vision","area_id":"computer-vision","collection":"Convolutions","url":"/methods/category/convolutions","pwc_aliases":[]}],"n_papers_tagged":3,"archive_num_papers":3,"papers_newest_first":[{"paper":"/paper/risurconv-rotation-invariant-surface","title":"RISurConv: Rotation Invariant Surface Attention-Augmented Convolutions for 3D Point Cloud Classification and Segmentation","date":"2024-08-12","arxiv_id":"2408.06110","n_code_links":1,"syntology":{"ran":11,"of":16,"unverified":5,"pointer_only":0}},{"paper":null,"title":"MuSLCAT: Multi-Scale Multi-Level Convolutional Attention Transformer for Discriminative Music Modeling on Raw Waveforms","date":"2021-04-06","arxiv_id":"2104.02309","n_code_links":0,"syntology":null},{"paper":"/paper/190409925","title":"Attention Augmented Convolutional Networks","date":"2019-04-22","arxiv_id":"1904.09925","n_code_links":14,"syntology":{"ran":3,"of":6,"unverified":3,"pointer_only":1}}],"papers_shown":3,"tasks":[{"task":"/task/3d-point-cloud-classification","name":"3D Point Cloud Classification","papers":1},{"task":"/task/classification","name":"General Classification","papers":1},{"task":"/task/image-classification","name":"Image Classification","papers":1},{"task":"/task/music-modeling","name":"Music Modeling","papers":1},{"task":"/task/music-tagging","name":"Music Tagging","papers":1},{"task":"/task/object-detection","name":"Object Detection","papers":1},{"task":"/task/point-cloud-classification","name":"Point Cloud Classification","papers":1},{"task":"/task/image-classification","name":"image-classification","papers":1},{"task":"/task/object-detection-1","name":"object-detection","papers":1}],"tasks_shown":9,"n_tasks":9,"usage_by_year":[{"year":"2019","papers":1},{"year":"2021","papers":1},{"year":"2024","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/attention-augmented-convolution"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}