Methods › General › Attention Mechanisms › Neighborhood Attention
Neighborhood Attention
Introduced by Ali Hassani et al. in Neighborhood Attention Transformer
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Neighborhood Attention is a restricted self attention pattern in which each token's receptive field is limited to its nearest neighboring pixels. It was proposed in Neighborhood Attention Transformer as an alternative to other local attention mechanisms used in Hierarchical Vision Transformers.
NA is in concept similar to stand alone self attention (SASA), in that both can be implemented with a raster scan sliding window operation over the key value pair. However, NA would require a modification to handle corner pixels, which helps maintain a fixed receptive field size and an increased number of relative positions.
The primary challenge in experimenting with both NA and SASA has been computation. Simply extracting key values for each query is slow, takes up a large amount of memory, and is eventually intractable at scale. NA was therefore implemented through a new CUDA extension to PyTorch, NATTEN.
Papers archive 2025-07-28
21 shown of 21, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Attention on the Sphere 16 May 2025 · 1 repository · arXiv:2505.11157
-
Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light 23 Apr 2025 · 1 repository · arXiv:2504.16922
-
Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention 19 Feb 2025 · 1 repository · arXiv:2502.13693
-
D3RM: A Discrete Denoising Diffusion Refinement Model for Piano Transcription 9 Jan 2025 · 1 repository · arXiv:2501.05068
-
PIGUIQA: A Physical Imaging Guided Perceptual Framework for Underwater Image Quality Assessment 20 Dec 2024 · 0 repositories · arXiv:2412.15527
-
EdgeNAT: Transformer for Efficient Edge Detection 20 Aug 2024 · 1 repository · arXiv:2408.10527
-
Compression-Realized Deep Structural Network for Video Quality Enhancement 10 May 2024 · 0 repositories · arXiv:2405.06342
-
A Point-Based Approach to Efficient LiDAR Multi-Task Perception 19 Apr 2024 · 0 repositories · arXiv:2404.12798
-
DeblurDiNAT: A Compact Model with Exceptional Generalization and Visual Fidelity on Unseen Domains 19 Mar 2024 · 1 repository · arXiv:2403.13163
-
Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level 7 Mar 2024 · 1 repository · arXiv:2403.04690Syntology ran 1 of 3 samples · 2 unverified
-
NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding 12 Dec 2023 · 1 repository · arXiv:2312.07507
-
SDLFormer: A Sparse and Dense Locality-enhanced Transformer for Accelerated MR Image Reconstruction 8 Aug 2023 · 1 repository · arXiv:2308.04262
-
ModeT: Learning Deformable Image Registration via Motion Decomposition Transformer 9 Jun 2023 · 1 repository · arXiv:2306.05688
-
Dilated-UNet: A Fast and Accurate Medical Image Segmentation Approach using a Dilated Transformer and U-Net Architecture 22 Apr 2023 · 1 repository · arXiv:2304.11450
-
Incorporating Transformer Designs into Convolutions for Lightweight Image Super-Resolution 25 Mar 2023 · 1 repository · arXiv:2303.14324
-
OneFormer: One Transformer to Rule Universal Image Segmentation 10 Nov 2022 · 4 repositories · arXiv:2211.06220Syntology ran 0 of 5 samples · 5 unverified
-
StyleNAT: Giving Each Head a New Perspective 10 Nov 2022 · 2 repositories · arXiv:2211.05770Syntology ran 0 of 1 samples · 1 unverified
-
Dilated Neighborhood Attention Transformer 29 Sep 2022 · 7 repositories · arXiv:2209.15001
-
V²L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval 26 Jul 2022 · 1 repository · arXiv:2207.12994
-
GAF-NAU: Gramian Angular Field encoded Neighborhood Attention U-Net for Pixel-Wise Hyperspectral Image Classification 21 Apr 2022 · 0 repositories · arXiv:2204.10099
-
Neighborhood Attention Transformer 14 Apr 2022 · 5 repositories · arXiv:2204.07143Syntology ran 1 of 1 samples · 0 unverified
Tasks archive 2025-07-28
20 shown of 47 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections