{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/davit-dual-attention-vision-transformers","title":"DaViT: Dual Attention Vision Transformers","arxiv_id":"2204.03645","date":"2022-04-07","proceeding":null,"authors":["Mingyu Ding","Bin Xiao","Noel Codella","Ping Luo","Jingdong Wang","Lu Yuan"],"abstract":"In this work, we introduce Dual Attention Vision Transformers (DaViT), a simple yet effective vision transformer architecture that is able to capture global context while maintaining computational efficiency. We propose approaching the problem from an orthogonal angle: exploiting self-attention mechanisms with both \"spatial tokens\" and \"channel tokens\". With spatial tokens, the spatial dimension defines the token scope, and the channel dimension defines the token feature dimension. With channel tokens, we have the inverse: the channel dimension defines the token scope, and the spatial dimension defines the token feature dimension. We further group tokens along the sequence direction for both spatial and channel tokens to maintain the linear complexity of the entire model. We show that these two self-attentions complement each other: (i) since each channel token contains an abstract representation of the entire image, the channel attention naturally captures global interactions and representations by taking all spatial positions into account when computing attention scores between channels; (ii) the spatial attention refines the local representations by performing fine-grained interactions across spatial locations, which in turn helps the global information modeling in channel attention. Extensive experiments show our DaViT achieves state-of-the-art performance on four different tasks with efficient computations. Without extra data, DaViT-Tiny, DaViT-Small, and DaViT-Base achieve 82.8%, 84.2%, and 84.6% top-1 accuracy on ImageNet-1K with 28.3M, 49.7M, and 87.9M parameters, respectively. When we further scale up DaViT with 1.5B weakly supervised image and text pairs, DaViT-Gaint reaches 90.4% top-1 accuracy on ImageNet-1K. Code is available at https://github.com/dingmyu/davit.","url_abs":"https://arxiv.org/abs/2204.03645v1","url_pdf":"https://arxiv.org/pdf/2204.03645v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"davit-dual-attention-vision-transformers","repo_url":"https://github.com/dingmyu/davit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"davit-dual-attention-vision-transformers","repo_url":"https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/davit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"davit-dual-attention-vision-transformers","repo_url":"https://github.com/rwightman/pytorch-image-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"davit-dual-attention-vision-transformers","repo_url":"https://gitlab.com/birder/birder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"medical-image-classification","task_name":"Medical Image Classification"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DaViT-G","rank_in_archive_order":4,"of":1060,"metrics":{"GFLOPs":"1038","Number of params":"1437M","Top 1 Accuracy":"90.4%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DaViT-H","rank_in_archive_order":5,"of":1060,"metrics":{"GFLOPs":"334","Number of params":"362M","Top 1 Accuracy":"90.2%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DaViT-L (ImageNet-22k)","rank_in_archive_order":85,"of":1060,"metrics":{"GFLOPs":"103","Number of params":"196.8M","Top 1 Accuracy":"87.5%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DaViT-B (ImageNet-22k)","rank_in_archive_order":119,"of":1060,"metrics":{"GFLOPs":"46.4","Number of params":"87.9M","Top 1 Accuracy":"86.9%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DaViT-B","rank_in_archive_order":306,"of":1060,"metrics":{"GFLOPs":"15.5","Number of params":"87.9M","Top 1 Accuracy":"84.6%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"DaViT-T","rank_in_archive_order":498,"of":1060,"metrics":{"Number of params":"28.3M","Top 1 Accuracy":"82.8%"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco-minival","task":"Instance Segmentation","dataset":"COCO minival","model":"DaViT-T (Mask R-CNN, 36 epochs)","rank_in_archive_order":56,"of":93,"metrics":{"mask AP":"44.3"},"uses_additional_data":false},{"leaderboard":"/sota/medical-image-classification-on-imagenet","task":"Medical Image Classification","dataset":"ImageNet","model":"DaViT-T","rank_in_archive_order":1,"of":2,"metrics":{"GFLOPs":"4.5"},"uses_additional_data":false},{"leaderboard":"/sota/medical-image-classification-on-imagenet","task":"Medical Image Classification","dataset":"ImageNet","model":"DaViT-S","rank_in_archive_order":2,"of":2,"metrics":{"GFLOPs":"8.8","Top 1 Accuracy":"84.2%"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco-minival","task":"Object Detection","dataset":"COCO minival","model":"DaViT-T (Mask R-CNN, 36 epochs)","rank_in_archive_order":82,"of":220,"metrics":{"box AP":"49.9"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"DaViT-B","rank_in_archive_order":132,"of":235,"metrics":{"Validation mIoU":"49.4"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"DaViT-T","rank_in_archive_order":178,"of":235,"metrics":{"Validation mIoU":"46.3"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k-val","task":"Semantic Segmentation","dataset":"ADE20K val","model":"DaViT-S (UperNet)","rank_in_archive_order":61,"of":95,"metrics":{"mIoU":"48.8"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k-val","task":"Semantic Segmentation","dataset":"ADE20K val","model":"DaViT-B (UperNet)","rank_in_archive_order":71,"of":95,"metrics":{"mIoU":"46.3"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/2204.03645","atlas_url":"https://app.syntology.ai/?focus=2204.03645","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2204.03645"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rwightman/pytorch-image-models","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leondgarse/keras_cv_attention_models/tree/main/keras_cv_attention_models/davit","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://gitlab.com/birder/birder","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dingmyu/davit","reach":null}],"summary":{"ran":6,"ran_fixture":1,"ran_draft_wrong":1,"unverified":7},"by_repo_kind":{"official":{"samples":15,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ae32983bf5de53c3","entry":"ChannelAttention","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae32983bf5de53c3"}},{"code_sha256_prefix":"98189454f1018bed","entry":"ChannelBlock","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"98189454f1018bed"}},{"code_sha256_prefix":"c6ff164380e31b6a","entry":"ConvPosEnc","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6ff164380e31b6a"}},{"code_sha256_prefix":"c1395a46164d140a","entry":"Mlp","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c1395a46164d140a"}},{"code_sha256_prefix":"dc8f008b6667e65a","entry":"PatchEmbed","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc8f008b6667e65a"}},{"code_sha256_prefix":"5091ab6c514932c4","entry":"WindowAttention","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5091ab6c514932c4"}},{"code_sha256_prefix":"80c66c535be6f491","entry":"window_partition","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80c66c535be6f491"}},{"code_sha256_prefix":"b83f31fb8b7c976a","entry":"window_reverse","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b83f31fb8b7c976a"}},{"code_sha256_prefix":"ceb98806b990f4e0","entry":"DaViT","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ceb98806b990f4e0"}},{"code_sha256_prefix":"5066418c2fb3b6c8","entry":"MySequential","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5066418c2fb3b6c8"}},{"code_sha256_prefix":"dad192f68c8848d9","entry":"SpatialBlock","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dad192f68c8848d9"}},{"code_sha256_prefix":"266d4f400940d7a0","entry":"_init_conv_weights","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"266d4f400940d7a0"}},{"code_sha256_prefix":"62d0b713fcda8506","entry":"_init_vit_weights","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62d0b713fcda8506"}},{"code_sha256_prefix":"ef7b648f5c9e3de2","entry":"lecun_normal_","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef7b648f5c9e3de2"}},{"code_sha256_prefix":"2f6bf4b4b7d2ba1c","entry":"variance_scaling_","repo":"dingmyu/davit","repo_kind":"official","path":"timm/models/davit.py","file_url":"https://github.com/dingmyu/davit/blob/HEAD/timm/models/davit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f6bf4b4b7d2ba1c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}