{"url":"/method/deformable-attention-module","slug":"deformable-attention-module","name":"Deformable Attention Module","full_name":"Deformable Attention Module","full_name_withheld":false,"description_markdown":"**Deformable Attention Module** is an attention module used in the [Deformable DETR](https://paperswithcode.com/method/deformable-detr) architecture, which seeks to overcome one issue base [Transformer attention](https://paperswithcode.com/method/scaled) in that it looks over all possible spatial locations. Inspired by [deformable convolution](https://paperswithcode.com/method/deformable-convolution), the deformable attention module only attends to a small set of key sampling points around a reference point, regardless of the spatial size of the feature maps. By assigning only a small fixed number of keys for each query, the issues of convergence and feature spatial resolution can be mitigated.\r\n\r\nGiven an input feature map $x \\in \\mathbb{R}^{C \\times H \\times W}$, let $q$ index a query element with content feature $\\mathbf{z}\\_{q}$ and a 2-d reference point $\\mathbf{p}\\_{q}$, the deformable attention feature is calculated by:\r\n\r\n$$ \\text{DeformAttn}\\left(\\mathbf{z}\\_{q}, \\mathbf{p}\\_{q}, \\mathbf{x}\\right)=\\sum\\_{m=1}^{M} \\mathbf{W}\\_{m}\\left[\\sum\\_{k=1}^{K} A\\_{m q k} \\cdot \\mathbf{W}\\_{m}^{\\prime} \\mathbf{x}\\left(\\mathbf{p}\\_{q}+\\Delta \\mathbf{p}\\_{m q k}\\right)\\right]\r\n$$\r\n\r\nwhere $m$ indexes the attention head, $k$ indexes the sampled keys, and $K$ is the total sampled key number $(K \\ll H W) . \\Delta p_{m q k}$ and $A_{m q k}$ denote the sampling offset and attention weight of the $k^{\\text {th }}$ sampling point in the $m^{\\text {th }}$ attention head, respectively. The scalar attention weight $A_{m q k}$ lies in the range $[0,1]$, normalized by $\\sum_{k=1}^{K} A_{m q k}=1 . \\Delta \\mathbf{p}_{m q k} \\in \\mathbb{R}^{2}$ are of 2-d real numbers with unconstrained range. As $p\\_{q}+\\Delta p\\_{m q k}$ is fractional, bilinear interpolation is applied as in Dai et al. (2017) in computing $\\mathbf{x}\\left(\\mathbf{p}\\_{q}+\\Delta \\mathbf{p}\\_{m q k}\\right)$. Both $\\Delta \\mathbf{p}\\_{m q k}$ and $A\\_{m q k}$ are obtained via linear projection over the query feature $z\\_{q} .$ In implementation, the query feature $z\\_{q}$ is fed to a linear projection operator of $3 M K$ channels, where the first $2 M K$ channels encode the sampling offsets $\\Delta p\\_{m q k}$, and the remaining $M K$ channels are fed to a softmax operator to obtain the attention weights $A\\_{m q k}$.","description_state":"present","introduced_year":null,"introduced_by":{"title":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","paper":"/paper/deformable-detr-deformable-transformers-for-1","first_author":"Xizhou Zhu","n_authors":6,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/deformable-detr-deformable-transformers-for-1"},"source":{"url":"https://arxiv.org/abs/2010.04159v4","title":"Deformable DETR: Deformable Transformers for End-to-End Object Detection","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Attention Modules","url":"/methods/category/attention-modules","pwc_aliases":[]}],"n_papers_tagged":42,"archive_num_papers":42,"papers_newest_first":[{"paper":"/paper/walnutdata-a-uav-remote-sensing-dataset-of","title":"WalnutData: A UAV Remote Sensing Dataset of Green Walnuts and Model Evaluation","date":"2025-02-27","arxiv_id":"2502.20092","n_code_links":1,"syntology":null},{"paper":null,"title":"Advancing SEM Based Nano-Scale Defect Analysis in Semiconductor Manufacturing for Advanced IC Nodes","date":"2024-09-06","arxiv_id":"2409.04310","n_code_links":0,"syntology":null},{"paper":"/paper/u-decn-end-to-end-underwater-object-detection","title":"U-DECN: End-to-End Underwater Object Detection ConvNet with Improved DeNoising Training","date":"2024-08-11","arxiv_id":"2408.05780","n_code_links":1,"syntology":null},{"paper":null,"title":"Fisher-aware Quantization for DETR Detectors with Critical-category Objectives","date":"2024-07-03","arxiv_id":"2407.03442","n_code_links":0,"syntology":null},{"paper":"/paper/knowledge-driven-subspace-fusion-and-gradient","title":"Knowledge-driven Subspace Fusion and Gradient Coordination for Multi-modal Learning","date":"2024-06-20","arxiv_id":"2406.13979","n_code_links":1,"syntology":null},{"paper":null,"title":"Understanding differences in applying DETR to natural and medical images","date":"2024-05-27","arxiv_id":"2405.17677","n_code_links":0,"syntology":null},{"paper":null,"title":"Infrared Adversarial Car Stickers","date":"2024-05-16","arxiv_id":"2405.09924","n_code_links":0,"syntology":null},{"paper":null,"title":"LDTR: Transformer-based Lane Detection with Anchor-chain Representation","date":"2024-03-21","arxiv_id":"2403.14354","n_code_links":0,"syntology":null},{"paper":"/paper/generative-region-language-pretraining-for","title":"Generative Region-Language Pretraining for Open-Ended Object Detection","date":"2024-03-15","arxiv_id":"2403.10191","n_code_links":1,"syntology":{"ran":8,"of":11,"unverified":3,"pointer_only":0}},{"paper":"/paper/hybrid-proposal-refiner-revisiting-detr","title":"Hybrid Proposal Refiner: Revisiting DETR Series from the Faster R-CNN Perspective","date":"2024-01-01","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":null,"title":"KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling","date":"2024-01-01","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/mono3dvg-3d-visual-grounding-in-monocular","title":"Mono3DVG: 3D Visual Grounding in Monocular Images","date":"2023-12-13","arxiv_id":"2312.08022","n_code_links":1,"syntology":{"ran":5,"of":7,"unverified":2,"pointer_only":7}},{"paper":"/paper/towards-few-annotation-learning-for-object","title":"Towards Few-Annotation Learning for Object Detection: Are Transformer-based Models More Efficient ?","date":"2023-10-30","arxiv_id":"2310.19936","n_code_links":1,"syntology":null},{"paper":"/paper/dac-detr-divide-the-attention-layers-and","title":"DAC-DETR: Divide the Attention Layers and Conquer","date":"2023-09-21","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/a-spatial-temporal-deformable-attention-based","title":"A Spatial-Temporal Deformable Attention based Framework for Breast Lesion Detection in Videos","date":"2023-09-09","arxiv_id":"2309.04702","n_code_links":1,"syntology":null},{"paper":null,"title":"Selecting Learnable Training Samples is All DETRs Need in Crowded Pedestrian Detection","date":"2023-05-18","arxiv_id":"2305.10801","n_code_links":0,"syntology":null},{"paper":null,"title":"Robust Traffic Light Detection Using Salience-Sensitive Loss: Computational Framework and Evaluations","date":"2023-05-08","arxiv_id":"2305.04516","n_code_links":0,"syntology":null},{"paper":null,"title":"Continual Detection Transformer for Incremental Object Detection","date":"2023-04-06","arxiv_id":"2304.03110","n_code_links":0,"syntology":null},{"paper":"/paper/vddt-improving-vessel-detection-with","title":"VDDT: Improving Vessel Detection with Deformable Transfomer","date":"2023-03-15","arxiv_id":null,"n_code_links":0,"syntology":null},{"paper":"/paper/ars-detr-aspect-ratio-sensitive-oriented","title":"ARS-DETR: Aspect Ratio-Sensitive Detection Transformer for Aerial Oriented Object Detection","date":"2023-03-09","arxiv_id":"2303.04989","n_code_links":1,"syntology":null},{"paper":null,"title":"Towards Robust Video Instance Segmentation with Temporal-Aware Transformer","date":"2023-01-20","arxiv_id":"2301.09416","n_code_links":0,"syntology":null},{"paper":null,"title":"Salient Sign Detection In Safe Autonomous Driving: AI Which Reasons Over Full Visual Context","date":"2023-01-14","arxiv_id":"2301.05804","n_code_links":0,"syntology":null},{"paper":"/paper/cross-modal-learning-with-3d-deformable","title":"Cross-Modal Learning with 3D Deformable Attention for Action Recognition","date":"2022-12-12","arxiv_id":"2212.05638","n_code_links":0,"syntology":null},{"paper":null,"title":"Open World DETR: Transformer based Open World Object Detection","date":"2022-12-06","arxiv_id":"2212.02969","n_code_links":0,"syntology":null},{"paper":"/paper/fqdet-fast-converging-query-based-detector","title":"FQDet: Fast-converging Query-based Detector","date":"2022-10-05","arxiv_id":"2210.02318","n_code_links":2,"syntology":{"ran":1,"of":6,"unverified":5,"pointer_only":3}},{"paper":null,"title":"ComplETR: Reducing the cost of annotations for object detection in dense scenes with vision transformers","date":"2022-09-13","arxiv_id":"2209.05654","n_code_links":0,"syntology":null},{"paper":"/paper/dptext-detr-towards-better-scene-text","title":"DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformer","date":"2022-07-10","arxiv_id":"2207.04491","n_code_links":1,"syntology":null},{"paper":"/paper/yolov7-trainable-bag-of-freebies-sets-new","title":"YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors","date":"2022-07-06","arxiv_id":"2207.02696","n_code_links":21,"syntology":{"ran":1,"of":11,"unverified":10,"pointer_only":0}},{"paper":"/paper/an-empirical-study-of-self-supervised","title":"An Empirical Study Of Self-supervised Learning Approaches For Object Detection With Transformers","date":"2022-05-11","arxiv_id":"2205.05543","n_code_links":2,"syntology":null},{"paper":"/paper/cross-domain-object-detection-with-mean","title":"MTTrans: Cross-Domain Object Detection with Mean-Teacher Transformer","date":"2022-05-03","arxiv_id":"2205.01643","n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/object-detection","name":"Object Detection","papers":28},{"task":"/task/object-detection-1","name":"object-detection","papers":23},{"task":"/task/object","name":"Object","papers":17},{"task":"/task/decoder","name":"Decoder","papers":10},{"task":"/task/instance-segmentation","name":"Instance Segmentation","papers":4},{"task":"/task/2d-object-detection","name":"2D Object Detection","papers":3},{"task":"/task/autonomous-driving","name":"Autonomous Driving","papers":3},{"task":null,"name":"GPU","papers":3},{"task":"/task/knowledge-distillation","name":"Knowledge Distillation","papers":3},{"task":"/task/pedestrian-detection","name":"Pedestrian Detection","papers":3},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":3},{"task":"/task/language-modeling","name":"Language Modeling","papers":2},{"task":"/task/language-modelling","name":"Language Modelling","papers":2},{"task":"/task/object-localization","name":"Object Localization","papers":2},{"task":"/task/optical-flow-estimation","name":"Optical Flow Estimation","papers":2},{"task":"/task/real-time-object-detection","name":"Real-Time Object Detection","papers":2},{"task":"/task/region-proposal","name":"Region Proposal","papers":2},{"task":"/task/segmentation","name":"Segmentation","papers":2},{"task":"/task/semi-supervised-object-detection","name":"Semi-Supervised Object Detection","papers":2},{"task":"/task/video-instance-segmentation","name":"Video Instance Segmentation","papers":2}],"tasks_shown":20,"n_tasks":66,"usage_by_year":[{"year":"2020","papers":1},{"year":"2021","papers":7},{"year":"2022","papers":12},{"year":"2023","papers":11},{"year":"2024","papers":10},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/deformable-attention-module"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}