Methods › General › Attention Modules › Spatially Separable Self-Attention
Spatially Separable Self-Attention
Introduced by Xiangxiang Chu et al. in Twins: Revisiting the Design of Spatial Attention in Vision Transformers
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Spatially Separable Self-Attention, or SSSA, is an attention module used in the Twins-SVT architecture that aims to reduce the computational complexity of vision transformers for dense prediction tasks (given high-resolution inputs). SSSA is composed of locally-grouped self-attention (LSA) and global sub-sampled attention (GSA).
Formally, spatially separable self-attention (SSSA) can be written as:
𝐳̂_(i j)ˡ= LSA ( LayerNorm (𝐳_(i j)ˡ⁻¹))+𝐳_(i j)ˡ⁻¹
𝐳_(i j)ˡ=FFN(LayerNorm(𝐳̂_(i j)ˡ))+𝐳̂_(i j)ˡ
𝐳̂ˡ⁺¹= GSA ( LayerNorm (𝐳ˡ))+𝐳ˡ
𝐳ˡ⁺¹= FFN ( LayerNorm (𝐳̂ˡ⁺¹))+𝐳̂ˡ⁺¹
i ∈{1,2, …., m}, j ∈{1,2, …., n}
where LSA means locally-grouped self-attention within a sub-window; GSA is the global sub-sampled attention by interacting with the representative keys (generated by the sub-sampling functions) from each sub-window 𝐳̂_(i j) ∈ℛ^(k₁ ×k₂ ×C) . Both LSA and GSA have multiple heads as in the standard self-attention.
Papers archive 2025-07-28
2 shown of 2, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Logically at Factify 2: A Multi-Modal Fact Checking System Based on Evidence Retrieval techniques and Transformer Encoder Architecture 9 Jan 2023 · 0 repositories · arXiv:2301.03127
-
Twins: Revisiting the Design of Spatial Attention in Vision Transformers 28 Apr 2021 · 9 repositories · arXiv:2104.13840Syntology ran 0 of 2 samples · 2 unverified · 2 pointer-only (licence)
Tasks archive 2025-07-28
7 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Avg | 1 |
| Benchmarking | 1 |
| Fact Checking | 1 |
| Fact Verification | 1 |
| Image Classification | 1 |
| Retrieval | 1 |
| Semantic Segmentation | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections