{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vision-transformer-with-deformable-attention","title":"Vision Transformer with Deformable Attention","arxiv_id":"2201.00520","date":"2022-01-03","proceeding":"CVPR 2022 1","authors":["Zhuofan Xia","Xuran Pan","Shiji Song","Li Erran Li","Gao Huang"],"abstract":"Transformers have recently shown superior performances on various vision tasks. The large, sometimes even global, receptive field endows Transformer models with higher representation power over their CNN counterparts. Nevertheless, simply enlarging receptive field also gives rise to several concerns. On the one hand, using dense attention e.g., in ViT, leads to excessive memory and computational cost, and features can be influenced by irrelevant parts which are beyond the region of interests. On the other hand, the sparse attention adopted in PVT or Swin Transformer is data agnostic and may limit the ability to model long range relations. To mitigate these issues, we propose a novel deformable self-attention module, where the positions of key and value pairs in self-attention are selected in a data-dependent way. This flexible scheme enables the self-attention module to focus on relevant regions and capture more informative features. On this basis, we present Deformable Attention Transformer, a general backbone model with deformable attention for both image classification and dense prediction tasks. Extensive experiments show that our models achieve consistently improved results on comprehensive benchmarks. Code is available at https://github.com/LeapLabTHU/DAT.","url_abs":"https://arxiv.org/abs/2201.00520v3","url_pdf":"https://arxiv.org/pdf/2201.00520v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vision-transformer-with-deformable-attention","repo_url":"https://github.com/leaplabthu/dat","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"vision-transformer-with-deformable-attention","repo_url":"https://github.com/ChristophReich1996/Swin-Transformer-V2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"pvt","method_name":"PVT"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"spatial-reduction-attention","method_name":"Spatial-Reduction Attention"},{"method_slug":"stochastic-depth","method_name":"Stochastic Depth"},{"method_slug":"swin-transformer","method_name":"Swin Transformer"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DAT-B (384 res, IN-1K only)","rank_in_archive_order":289,"of":1060,"metrics":{"GFLOPs":"49.8","Number of params":"88M","Top 1 Accuracy":"84.8%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DAT-S","rank_in_archive_order":396,"of":1060,"metrics":{"GFLOPs":"9.0","Number of params":"50M","Top 1 Accuracy":"83.7%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DAT-T","rank_in_archive_order":588,"of":1060,"metrics":{"GFLOPs":"4.6","Number of params":"29M","Top 1 Accuracy":"82.0%"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"DAT-S (RetinaNet)","rank_in_archive_order":112,"of":225,"metrics":{"AP50":"69.6","AP75":"51.2","APL":"63.4","APM":"51.8","APS":"32.3","box mAP":"47.9"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"DAT-B (UperNet)","rank_in_archive_order":133,"of":235,"metrics":{"Params (M)":"121","Validation mIoU":"49.38"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"DAT-S (UperNet)","rank_in_archive_order":148,"of":235,"metrics":{"Params (M)":"81","Validation mIoU":"48.31"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"DAT-T (UperNet)","rank_in_archive_order":190,"of":235,"metrics":{"Params (M)":"60","Validation mIoU":"45.54"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2201.00520","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2201.00520"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ChristophReich1996/Swin-Transformer-V2","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leaplabthu/dat","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/LeapLabTHU/DAT","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":4,"ran_draft_wrong":3,"ran_fixture":1,"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7cba8d74ab462cab","entry":"DAttentionBaseline","repo":"LeapLabTHU/DAT","repo_kind":"official","path":"models/dat_blocks.py","file_url":"https://github.com/LeapLabTHU/DAT/blob/HEAD/models/dat_blocks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7cba8d74ab462cab"}},{"code_sha256_prefix":"2aa71e543f03992a","entry":"FeedForward","repo":"ChristophReich1996/Swin-Transformer-V2","repo_kind":"listed","path":"swin_transformer_v2/model_parts.py","file_url":"https://github.com/ChristophReich1996/Swin-Transformer-V2/blob/HEAD/swin_transformer_v2/model_parts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2aa71e543f03992a"}},{"code_sha256_prefix":"5ebf85fee0f0738f","entry":"LayerNormProxy","repo":"LeapLabTHU/DAT","repo_kind":"official","path":"models/dat_blocks.py","file_url":"https://github.com/LeapLabTHU/DAT/blob/HEAD/models/dat_blocks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ebf85fee0f0738f"}},{"code_sha256_prefix":"5716ef7256430436","entry":"WindowMultiHeadAttention","repo":"ChristophReich1996/Swin-Transformer-V2","repo_kind":"listed","path":"swin_transformer_v2/model_parts.py","file_url":"https://github.com/ChristophReich1996/Swin-Transformer-V2/blob/HEAD/swin_transformer_v2/model_parts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5716ef7256430436"}},{"code_sha256_prefix":"f76dfdc07001b5e3","entry":"bchw_to_bhwc","repo":"ChristophReich1996/Swin-Transformer-V2","repo_kind":"listed","path":"swin_transformer_v2/model_parts.py","file_url":"https://github.com/ChristophReich1996/Swin-Transformer-V2/blob/HEAD/swin_transformer_v2/model_parts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f76dfdc07001b5e3"}},{"code_sha256_prefix":"8c3d7c2fece98766","entry":"bhwc_to_bchw","repo":"ChristophReich1996/Swin-Transformer-V2","repo_kind":"listed","path":"swin_transformer_v2/model_parts.py","file_url":"https://github.com/ChristophReich1996/Swin-Transformer-V2/blob/HEAD/swin_transformer_v2/model_parts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c3d7c2fece98766"}},{"code_sha256_prefix":"ae92018684b30199","entry":"fold","repo":"ChristophReich1996/Swin-Transformer-V2","repo_kind":"listed","path":"swin_transformer_v2/model_parts.py","file_url":"https://github.com/ChristophReich1996/Swin-Transformer-V2/blob/HEAD/swin_transformer_v2/model_parts.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae92018684b30199"}},{"code_sha256_prefix":"31576ed450a2bd34","entry":"unfold","repo":"ChristophReich1996/Swin-Transformer-V2","repo_kind":"listed","path":"swin_transformer_v2/model_parts.py","file_url":"https://github.com/ChristophReich1996/Swin-Transformer-V2/blob/HEAD/swin_transformer_v2/model_parts.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31576ed450a2bd34"}},{"code_sha256_prefix":"6fba747e5af83c5f","entry":"DeformableSwinTransformerBlock","repo":"ChristophReich1996/Swin-Transformer-V2","repo_kind":"listed","path":"swin_transformer_v2/model_parts.py","file_url":"https://github.com/ChristophReich1996/Swin-Transformer-V2/blob/HEAD/swin_transformer_v2/model_parts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6fba747e5af83c5f"}},{"code_sha256_prefix":"3b8f635fca41be47","entry":"SwinTransformerBlock","repo":"ChristophReich1996/Swin-Transformer-V2","repo_kind":"listed","path":"swin_transformer_v2/model_parts.py","file_url":"https://github.com/ChristophReich1996/Swin-Transformer-V2/blob/HEAD/swin_transformer_v2/model_parts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b8f635fca41be47"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}