Methods › General › Attention Modules › Spatial-Reduction Attention
Spatial-Reduction Attention
Introduced by Wenhai Wang et al. in Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Spatial-Reduction Attention, or SRA, is a multi-head attention module used in the Pyramid Vision Transformer architecture which reduces the spatial scale of the key K and value V before the attention operation. This reduces the computational/memory overhead. Details of the SRA in the stage i can be formulated as follows:
SRA(Q, K, V)= Concat (head₀, … head _(Nᵢ)) Wᴼ
headⱼ= Attention (Q Wⱼ^Q, SR(K) Wⱼᴷ, SR(V) Wⱼⱽ)
where Concat (·) is the concatenation operation. Wⱼ^Q ∈ℝ^(Cᵢ ×d_(head)), Wⱼᴷ ∈ℝ^(Cᵢ ×d_(head)), Wⱼⱽ ∈ℝ^(Cᵢ ×d_(head)), and Wᴼ ∈ℝ^(Cᵢ ×Cᵢ) are linear projection parameters. Nᵢ is the head number of the attention layer in Stage i. Therefore, the dimension of each head (i.e. .d_(head)) is equal to Cᵢ/Nᵢ . SR(·) is the operation for reducing the spatial dimension of the input sequence (K or V ), which is written as:
SR(x)=Norm(Reshape(x, Rᵢ) W^S)
Here, x ∈ℝ^((Hᵢ Wᵢ) ×Cᵢ) represents a input sequence, and Rᵢ denotes the reduction ratio of the attention layers in Stage i . Reshape (x, Rᵢ) is an operation of reshaping the input sequence x to a sequence of size (Hᵢ Wᵢ)/Rᵢ² ×(Rᵢ² Cᵢ). W_S ∈ℝ^((Rᵢ² Cᵢ) ×Cᵢ) is a linear projection that reduces the dimension of the input sequence to Cᵢ. Norm(·) refers to layer normalization.
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: geospatial attention; Future Spatial Attention (FSA); Spatial-Aware Reduction Framework; Spatial-Aware Reduction Framework; Low-Rank Spatial Attention (LRSA); Contextual Spatial Attention (CSA); IB-inspired spatial attention module; SpatialFormer Target Attention (SFTA).
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 29 · with a code link: 13 · where Syntology ran a sample: 5 (5 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (5 of 29 tagged: 5 with a run with no instrument failure, 0 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 29 of 29, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
GLOVA: Global and Local Variation-Aware Analog Circuit Design with Risk-Sensitive Reinforcement Learning 16 May 2025 · 0 repositories · arXiv:2505.11208
-
Crystal Oscillators in OSNMA-Enabled Receivers: An Implementation View for Automotive Applications 25 Jan 2025 · 0 repositories · arXiv:2501.15123
-
Multipath Mitigation Technology-integrated GNSS Direct Position Estimation Plug-in Module 20 Nov 2024 · 0 repositories · arXiv:2411.13339
-
HRPVT: High-Resolution Pyramid Vision Transformer for medium and small-scale human pose estimation 29 Oct 2024 · 0 repositories · arXiv:2410.22079
-
Low-Rank Continual Pyramid Vision Transformer: Incrementally Segment Whole-Body Organs in CT with Light-Weighted Adaptation 7 Oct 2024 · 0 repositories · arXiv:2410.04689
-
Recasting Generic Pretrained Vision Transformers As Object-Centric Scene Encoders For Manipulation Policies 24 May 2024 · 0 repositories · arXiv:2405.15916
-
Rethinking Attention Gated with Hybrid Dual Pyramid Transformer-CNN for Generalized Segmentation in Medical Imaging 28 Apr 2024 · 1 repository · arXiv:2404.18199
-
Multi-Layer Dense Attention Decoder for Polyp Segmentation 27 Mar 2024 · 1 repository · arXiv:2403.18180
-
Heracles: A Hybrid SSM-Transformer Model for High-Resolution Image and Time-Series Analysis 26 Mar 2024 · 2 repositories · arXiv:2403.18063Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 1 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
ROI-Aware Multiscale Cross-Attention Vision Transformer for Pest Image Identification 28 Dec 2023 · 0 repositories · arXiv:2312.16914
-
Distilling Knowledge from CNN-Transformer Models for Enhanced Human Action Recognition 2 Nov 2023 · 0 repositories · arXiv:2311.01283
-
SeUNet-Trans: A Simple yet Effective UNet-Transformer Model for Medical Image Segmentation 16 Oct 2023 · 0 repositories · arXiv:2310.09998
-
DAT++: Spatially Dynamic Vision Transformer with Deformable Attention 4 Sep 2023 · 1 repository · arXiv:2309.01430Syntology official (archive's flag): 8 ran · 8 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 1 violated, 4 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 10 harvested samples) · 6 pointer-only (licence)
-
A denoised Mean Teacher for domain adaptive point cloud registration 26 Jun 2023 · 1 repository · arXiv:2306.14749
-
A 3-step Low-latency Low-Power Multichannel Time-to-Digital Converter based on Time Residual Amplifier 1 Jun 2023 · 0 repositories · arXiv:2306.00433
-
Neural correlates of cognitive ability and visuo-motor speed: validation of IDoCT on UK Biobank Data 30 May 2023 · 0 repositories · arXiv:2305.18804
-
PVT-SSD: Single-Stage 3D Object Detector with Point-Voxel Transformer 11 May 2023 · 1 repository · arXiv:2305.06621
-
Sector Patch Embedding: An Embedding Module Conforming to The Distortion Pattern of Fisheye Image 26 Mar 2023 · 0 repositories · arXiv:2303.14645
-
Chasing Clouds: Differentiable Volumetric Rasterisation of Point Clouds as a Highly Efficient and Accurate Loss for Large-Scale Deformable 3D Registration 1 Jan 2023 · 1 repository
-
Exploring the Relationship Between Architectural Design and Adversarially Robust Generalization 1 Jan 2023 · 0 repositories
-
EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones 17 Nov 2022 · 1 repository · arXiv:2211.09703
-
Exploring the Relationship between Architecture and Adversarially Robust Generalization 28 Sep 2022 · 0 repositories · arXiv:2209.14105
-
Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality 20 May 2022 · 1 repository · arXiv:2205.10063
-
Uncertainty-Cognizant Model Predictive Control for Energy Management of Residential Buildings with PVT and Thermal Energy Storage 21 Jan 2022 · 0 repositories · arXiv:2201.08909
-
Vision Transformer with Deformable Attention 3 Jan 2022 · 2 repositories · arXiv:2201.00520Syntology official (archive's flag): 2 ran · 8 ran (of which 4 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 10 harvested samples)
-
Dynamic Token Normalization Improves Vision Transformers 5 Dec 2021 · 1 repository · arXiv:2112.02624Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
-
Partial Variable Training for Efficient On-Device Federated Learning 11 Oct 2021 · 0 repositories · arXiv:2110.05607
-
PVT: Point-Voxel Transformer for Point Cloud Learning 13 Aug 2021 · 2 repositories · arXiv:2108.06076
-
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions 24 Feb 2021 · 11 repositories · arXiv:2102.12122Syntology official (archive's flag): 4 ran · 22 ran (of which 16 constructed an object rather than computing a result; 18 with no instrument failure: 2 honoured, 0 violated, 16 with no contract checked; 4 where Syntology's instrument failed) · 8 unverified (of 30 harvested samples) · 1 pointer-only (licence)
Tasks archive 2025-07-28
20 shown of 43 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections