{"url":"/method/locally-grouped-self-attention","slug":"locally-grouped-self-attention","name":"Locally-Grouped Self-Attention","full_name":"Locally-Grouped Self-Attention","full_name_withheld":false,"description_markdown":"**Locally-Grouped Self-Attention**, or **LSA**, is a local attention mechanism used in the [Twins-SVT](https://paperswithcode.com/method/twins-svt) architecture. Locally-grouped self-attention (LSA). Motivated by the group design in depthwise convolutions for efficient inference, we first equally divide the 2D feature maps into sub-windows, making self-attention communications only happen within each sub-window. This design also resonates with the multi-head design in self-attention, where the communications only occur within the channels of the same head. To be specific, the feature maps are divided into $m \\times n$ sub-windows. Without loss of generality, we assume $H \\% m=0$ and $W \\% n=0$. Each group contains $\\frac{H W}{m n}$ elements, and thus the computation cost of the self-attention in this window is $\\mathcal{O}\\left(\\frac{H^{2} W^{2}}{m^{2} n^{2}} d\\right)$, and the total cost is $\\mathcal{O}\\left(\\frac{H^{2} W^{2}}{m n} d\\right)$. If we let $k\\_{1}=\\frac{H}{n}$ and $k\\_{2}=\\frac{W}{n}$, the cost can be computed as $\\mathcal{O}\\left(k\\_{1} k\\_{2} H W d\\right)$, which is significantly more efficient when $k\\_{1} \\ll H$ and $k\\_{2} \\ll W$ and grows linearly with $H W$ if $k\\_{1}$ and $k\\_{2}$ are fixed.\r\n\r\nAlthough the locally-grouped self-attention mechanism is computation friendly, the image is divided into non-overlapping sub-windows. Thus, we need a mechanism to communicate between different sub-windows, as in Swin. Otherwise, the information would be limited to be processed locally, which makes the receptive field small and significantly degrades the performance as shown in our experiments. This resembles the fact that we cannot replace all standard convolutions by depth-wise convolutions in CNNs.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Twins: Revisiting the Design of Spatial Attention in Vision Transformers","paper":"/paper/twins-revisiting-spatial-attention-design-in","first_author":"Xiangxiang Chu","n_authors":8,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/twins-revisiting-spatial-attention-design-in"},"source":{"url":"https://arxiv.org/abs/2104.13840v4","title":"Twins: Revisiting the Design of Spatial Attention in Vision Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Mechanisms","url":"/methods/category/attention-mechanisms","pwc_aliases":["attention-mechanisms-1"]}],"n_papers_tagged":2,"archive_num_papers":2,"papers_newest_first":[{"paper":null,"title":"Logically at Factify 2: A Multi-Modal Fact Checking System Based on Evidence Retrieval techniques and Transformer Encoder Architecture","date":"2023-01-09","arxiv_id":"2301.03127","n_code_links":0,"syntology":null},{"paper":"/paper/twins-revisiting-spatial-attention-design-in","title":"Twins: Revisiting the Design of Spatial Attention in Vision Transformers","date":"2021-04-28","arxiv_id":"2104.13840","n_code_links":9,"syntology":{"ran":0,"of":2,"unverified":2,"pointer_only":2}}],"papers_shown":2,"tasks":[{"task":null,"name":"Avg","papers":1},{"task":"/task/benchmarking","name":"Benchmarking","papers":1},{"task":"/task/fact-checking","name":"Fact Checking","papers":1},{"task":"/task/fact-verification","name":"Fact Verification","papers":1},{"task":"/task/image-classification","name":"Image Classification","papers":1},{"task":"/task/retrieval","name":"Retrieval","papers":1},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":1}],"tasks_shown":7,"n_tasks":7,"usage_by_year":[{"year":"2021","papers":1},{"year":"2023","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/locally-grouped-self-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}