{"url":"/method/split-attention","slug":"split-attention","name":"Split Attention","full_name":"Split Attention","full_name_withheld":false,"description_markdown":"A **Split Attention** block enables attention across feature-map groups. As in [ResNeXt blocks](https://paperswithcode.com/method/resnext-block), the feature can be divided into several groups, and the number of feature-map groups is given by a cardinality hyperparameter $K$. The resulting feature-map groups are called cardinal groups. Split Attention blocks introduce a new radix hyperparameter $R$ that indicates the number of splits within a cardinal group, so the total number of feature groups is $G = KR$. We may apply a series of transformations {$\\mathcal{F}\\_1, \\mathcal{F}\\_2, \\cdots\\mathcal{F}\\_G$} to each individual group, then the intermediate representation of each group is $U\\_i = \\mathcal{F}\\_i\\left(X\\right)$, for $i \\in$ {$1, 2, \\cdots{G}$}.\r\n\r\nA combined representation for each cardinal group can be obtained by fusing via an element-wise summation across multiple splits. The representation for $k$-th cardinal group is \r\n$\\hat{U}^k = \\sum_{j=R(k-1)+1}^{R k} U_j $, where $\\hat{U}^k \\in \\mathbb{R}^{H\\times W\\times C/K}$ for $k\\in{1,2,...K}$, and $H$, $W$ and $C$ are the block output feature-map sizes. \r\nGlobal contextual information with embedded channel-wise statistics can be gathered with [global average pooling](https://paperswithcode.com/method/global-average-pooling) across spatial dimensions  $s^k\\in\\mathbb{R}^{C/K}$. Here the $c$-th component is calculated as:\r\n\r\n$$\r\n    s^k\\_c = \\frac{1}{H\\times W} \\sum\\_{i=1}^H\\sum\\_{j=1}^W \\hat{U}^k\\_c(i, j).\r\n$$\r\n\r\nA weighted fusion of the cardinal group representation $V^k\\in\\mathbb{R}^{H\\times W\\times C/K}$ is aggregated using [channel-wise soft attention](https://paperswithcode.com/method/channel-wise-soft-attention), where each feature-map channel is produced using a weighted combination over splits. The $c$-th channel is calculated as:\r\n\r\n$$\r\n    V^k_c=\\sum_{i=1}^R a^k_i(c) U_{R(k-1)+i} ,\r\n$$\r\n\r\nwhere $a_i^k(c)$ denotes a (soft) assignment weight given by:\r\n\r\n$$\r\na_i^k(c) =\r\n\\begin{cases}\r\n  \\frac{exp(\\mathcal{G}^c_i(s^k))}{\\sum_{j=0}^R exp(\\mathcal{G}^c_j(s^k))} & \\quad\\textrm{if } R>1, \\\\\r\n   \\frac{1}{1+exp(-\\mathcal{G}^c_i(s^k))} & \\quad\\textrm{if } R=1,\\\\\r\n\\end{cases}\r\n$$\r\n\r\nand mapping $\\mathcal{G}_i^c$ determines the weight of each split for the $c$-th channel based on the global context representation $s^k$.","description_state":"present","introduced_year":null,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":"https://arxiv.org/abs/2004.08955v2","title":"ResNeSt: Split-Attention Networks","url_on_a_paper_host":true},"code_snippet_url":"https://github.com/zhanghang1989/ResNeSt/blob/5fe47e93bd7e098d15bc278d8ab4812b82b49414/resnest/torch/splat.py#L11","code_snippet_url_on_a_code_host":true,"categories":[{"area":"Computer Vision","area_id":"computer-vision","collection":"Image Model Blocks","url":"/methods/category/image-model-blocks","pwc_aliases":[]}],"n_papers_tagged":20,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/gaf-fusionnet-multimodal-ecg-analysis-via","title":"GAF-FusionNet: Multimodal ECG Analysis via Gramian Angular Fields and Split Attention","date":"2024-12-07","arxiv_id":"2501.01960","n_code_links":1,"syntology":null},{"paper":null,"title":"Efficiently Expanding Receptive Fields: Local Split Attention and Parallel Aggregation for Enhanced Large-scale Point Cloud Semantic Segmentation","date":"2024-09-03","arxiv_id":"2409.01662","n_code_links":0,"syntology":null},{"paper":null,"title":"Fusion Flow-enhanced Graph Pooling Residual Networks for Unmanned Aerial Vehicles Surveillance in Day and Night Dual Visions","date":"2024-07-17","arxiv_id":"2407.12647","n_code_links":0,"syntology":null},{"paper":null,"title":"Supervised domain adaptation for building extraction from off-nadir aerial images","date":"2023-11-07","arxiv_id":"2311.03867","n_code_links":0,"syntology":null},{"paper":null,"title":"Effect of roundabout design on the behavior of road users: A case study of roundabouts with application of Unsupervised Machine Learning","date":"2023-09-25","arxiv_id":"2309.14540","n_code_links":0,"syntology":null},{"paper":"/paper/tiefake-title-text-similarity-and-emotion","title":"TieFake: Title-Text Similarity and Emotion-Aware Fake News Detection","date":"2023-04-19","arxiv_id":"2304.09421","n_code_links":2,"syntology":null},{"paper":null,"title":"Early detection of hip periprosthetic joint infections through CNN on Computed Tomography images","date":"2023-04-18","arxiv_id":"2304.08942","n_code_links":0,"syntology":null},{"paper":null,"title":"OMSN and FAROS: OCTA Microstructure Segmentation Network and Fully Annotated Retinal OCTA Segmentation Dataset","date":"2022-12-26","arxiv_id":"2212.13059","n_code_links":0,"syntology":null},{"paper":null,"title":"Anomaly Detection in Automatic Generation Control Systems Based on Traffic Pattern Analysis and Deep Transfer Learning","date":"2022-09-16","arxiv_id":"2209.08099","n_code_links":0,"syntology":null},{"paper":"/paper/resnests-and-densenests-block-based-dnn","title":"ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees","date":"2021-11-10","arxiv_id":"2111.05496","n_code_links":4,"syntology":null},{"paper":"/paper/solution-for-large-scale-long-tailed","title":"Solution for Large-scale Long-tailed Recognition with Noisy Labels","date":"2021-06-20","arxiv_id":"2106.10683","n_code_links":1,"syntology":null},{"paper":"/paper/epsanet-an-efficient-pyramid-split-attention","title":"EPSANet: An Efficient Pyramid Squeeze Attention Block on Convolutional Neural Network","date":"2021-05-30","arxiv_id":"2105.14447","n_code_links":4,"syntology":null},{"paper":"/paper/ag-curesnest-a-novel-method-for-colon-polyp","title":"AG-CUResNeSt: A Novel Method for Colon Polyp Segmentation","date":"2021-05-02","arxiv_id":"2105.00402","n_code_links":1,"syntology":null},{"paper":null,"title":"Searching for Fast Model Families on Datacenter Accelerators","date":"2021-02-10","arxiv_id":"2102.05610","n_code_links":0,"syntology":null},{"paper":"/paper/bottleneck-transformers-for-visual","title":"Bottleneck Transformers for Visual Recognition","date":"2021-01-27","arxiv_id":"2101.11605","n_code_links":13,"syntology":{"ran":26,"of":49,"unverified":23,"pointer_only":8}},{"paper":"/paper/msaf-multimodal-split-attention-fusion","title":"MSAF: Multimodal Split Attention Fusion","date":"2020-12-13","arxiv_id":"2012.07175","n_code_links":1,"syntology":null},{"paper":null,"title":"Causal Contextual Prediction for Learned Image Compression","date":"2020-11-19","arxiv_id":"2011.09704","n_code_links":0,"syntology":null},{"paper":"/paper/face-hallucination-using-split-attention-in","title":"Face Hallucination via Split-Attention in Split-Attention Network","date":"2020-10-22","arxiv_id":"2010.11575","n_code_links":1,"syntology":null},{"paper":null,"title":"2nd Place Solution to Instance Segmentation of IJCAI 3D AI Challenge 2020","date":"2020-10-21","arxiv_id":"2010.10957","n_code_links":0,"syntology":null},{"paper":"/paper/resnest-split-attention-networks","title":"ResNeSt: Split-Attention Networks","date":"2020-04-19","arxiv_id":"2004.08955","n_code_links":36,"syntology":{"ran":8,"of":48,"unverified":40,"pointer_only":23}}],"papers_shown":20,"tasks":[{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":5},{"task":"/task/instance-segmentation","name":"Instance Segmentation","papers":4},{"task":"/task/segmentation","name":"Segmentation","papers":4},{"task":"/task/image-classification","name":"Image Classification","papers":3},{"task":"/task/object-detection","name":"Object Detection","papers":3},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":3},{"task":"/task/image-classification","name":"image-classification","papers":3},{"task":"/task/object-detection-1","name":"object-detection","papers":2},{"task":"/task/action-recognition-in-videos","name":"Action Recognition","papers":1},{"task":"/task/anomaly-detection","name":"Anomaly Detection","papers":1},{"task":"/task/articles","name":"Articles","papers":1},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":1},{"task":"/task/decoder","name":"Decoder","papers":1},{"task":"/task/domain-adaptation","name":"Domain Adaptation","papers":1},{"task":"/task/ecg-classification","name":"ECG Classification","papers":1},{"task":"/task/earth-observation","name":"Earth Observation","papers":1},{"task":"/task/emotion-recognition","name":"Emotion Recognition","papers":1},{"task":"/task/experimental-design","name":"Experimental Design","papers":1},{"task":"/task/face-detection","name":"Face Detection","papers":1},{"task":"/task/face-hallucination","name":"Face Hallucination","papers":1}],"tasks_shown":20,"n_tasks":45,"usage_by_year":[{"year":"2020","papers":5},{"year":"2021","papers":6},{"year":"2022","papers":2},{"year":"2023","papers":4},{"year":"2024","papers":3}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/split-attention"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}