{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/focal-self-attention-for-local-global","title":"Focal Self-attention for Local-Global Interactions in Vision Transformers","arxiv_id":"2107.00641","date":"2021-07-01","proceeding":null,"authors":["Jianwei Yang","Chunyuan Li","Pengchuan Zhang","Xiyang Dai","Bin Xiao","Lu Yuan","Jianfeng Gao"],"abstract":"Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing short- and long-range visual dependencies through self-attention is arguably the main source for the success. But it also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). In this paper, we present focal self-attention, a new mechanism that incorporates both fine-grained local and coarse-grained global interactions. Using this new mechanism, each token attends the closest surrounding tokens at fine granularity but the tokens far away at coarse granularity, and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks. In particular, our Focal Transformer models with a moderate size of 51.1M and a larger size of 89.8M achieve 83.5 and 83.8 Top-1 accuracy, respectively, on ImageNet classification at 224x224 resolution. Using Focal Transformers as the backbones, we obtain consistent and substantial improvements over the current state-of-the-art Swin Transformers for 6 different object detection methods trained with standard 1x and 3x schedules. Our largest Focal Transformer yields 58.7/58.9 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks.","url_abs":"https://arxiv.org/abs/2107.00641v1","url_pdf":"https://arxiv.org/pdf/2107.00641v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"focal-self-attention-for-local-global","repo_url":"https://github.com/microsoft/Focal-Transformer","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"focal-self-attention-for-local-global","repo_url":"https://github.com/microsoft/esvit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"focal-self-attention-for-local-global","repo_url":"https://github.com/BR-IDL/PaddleViT/tree/develop/image_classification/Focal_Transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"paddle","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"instance-segmentation","task_name":"Instance Segmentation"},{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focal-transformers","method_name":"Focal Transformers"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"vision-transformer","method_name":"Vision Transformer"}],"datasets_introduced":[],"methods_introduced":[{"slug":"focal-transformers","name":"Focal Transformers","full_name":"Focal Transformers"}],"results":[{"leaderboard":"/sota/instance-segmentation-on-coco-minival","task":"Instance Segmentation","dataset":"COCO minival","model":"Focal-L (HTC++, multi-scale)","rank_in_archive_order":21,"of":93,"metrics":{"mask AP":"50.9"},"uses_additional_data":false},{"leaderboard":"/sota/instance-segmentation-on-coco","task":"Instance Segmentation","dataset":"COCO test-dev","model":"Focal-L (HTC++, multi-scale)","rank_in_archive_order":19,"of":112,"metrics":{"AP50":"75.4","AP75":"56.5","APL":"64.2","APS":"35.6","mask AP":"51.3"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco-minival","task":"Object Detection","dataset":"COCO minival","model":"Focal-L (DyHead, multi-scale)","rank_in_archive_order":33,"of":220,"metrics":{"AP50":"77.2","APL":"73.4","box AP":"58.7"},"uses_additional_data":false},{"leaderboard":"/sota/object-detection-on-coco","task":"Object Detection","dataset":"COCO test-dev","model":"Focal-L (DyHead, multi-scale)","rank_in_archive_order":33,"of":225,"metrics":{"box mAP":"58.9"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k","task":"Semantic Segmentation","dataset":"ADE20K","model":"Focal-L (UperNet, ImageNet-22k pretrain)","rank_in_archive_order":47,"of":235,"metrics":{"Validation mIoU":"55.40"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-segmentation-on-ade20k-val","task":"Semantic Segmentation","dataset":"ADE20K val","model":"Focal-L (UperNet, ImageNet-22k pretrain)","rank_in_archive_order":30,"of":95,"metrics":{"mIoU":"55.4"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2107.00641","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2107.00641"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/esvit","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BR-IDL/PaddleViT/tree/develop/image_classification/Focal_Transformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/microsoft/Focal-Transformer","reach":null}],"summary":{"ran_fixture":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"144d10b49baeb8a6","entry":"window_partition","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"144d10b49baeb8a6"}},{"code_sha256_prefix":"4991d38b4d05d388","entry":"window_partition_noreshape","repo":"microsoft/Focal-Transformer","repo_kind":"official","path":"classification/focal_transformer.py","file_url":"https://github.com/microsoft/Focal-Transformer/blob/HEAD/classification/focal_transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4991d38b4d05d388"}},{"code_sha256_prefix":"61bf152e6a42a184","entry":"window_reverse","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"61bf152e6a42a184"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}