{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stand-alone-self-attention-in-vision-models","title":"Stand-Alone Self-Attention in Vision Models","arxiv_id":"1906.05909","date":"2019-06-13","proceeding":"NeurIPS 2019 12","authors":["Prajit Ramachandran","Niki Parmar","Ashish Vaswani","Irwan Bello","Anselm Levskaya","Jonathon Shlens"],"abstract":"Convolutions are a fundamental building block of modern computer vision systems. Recent approaches have argued for going beyond convolutions in order to capture long-range dependencies. These efforts focus on augmenting convolutional models with content-based interactions, such as self-attention and non-local means, to achieve gains on a number of vision tasks. The natural question that arises is whether attention can be a stand-alone primitive for vision models instead of serving as just an augmentation on top of convolutions. In developing and testing a pure self-attention vision model, we verify that self-attention can indeed be an effective stand-alone layer. A simple procedure of replacing all instances of spatial convolutions with a form of self-attention applied to ResNet model produces a fully self-attentional model that outperforms the baseline on ImageNet classification with 12% fewer FLOPS and 29% fewer parameters. On COCO object detection, a pure self-attention model matches the mAP of a baseline RetinaNet while having 39% fewer FLOPS and 34% fewer parameters. Detailed ablation studies demonstrate that self-attention is especially impactful when used in later layers. These results establish that stand-alone self-attention is an important addition to the vision practitioner's toolbox.","url_abs":"https://arxiv.org/abs/1906.05909v1","url_pdf":"https://arxiv.org/pdf/1906.05909v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stand-alone-self-attention-in-vision-models","repo_url":"https://github.com/google-research/google-research","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"stand-alone-self-attention-in-vision-models","repo_url":"https://github.com/Doyosae/Stand-Alone-Self-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"stand-alone-self-attention-in-vision-models","repo_url":"https://github.com/JoeRoussy/adaptive-attention-in-cv","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"stand-alone-self-attention-in-vision-models","repo_url":"https://github.com/MaheepChaudhary/Stand-Alone_Self-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"stand-alone-self-attention-in-vision-models","repo_url":"https://github.com/MartinGer/Stand-Alone-Self-Attention-in-Vision-Models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"stand-alone-self-attention-in-vision-models","repo_url":"https://github.com/leaderj1001/Stand-Alone-Self-Attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"stand-alone-self-attention-in-vision-models","repo_url":"https://github.com/The-AI-Summer/self_attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"stand-alone-self-attention-in-vision-models","repo_url":"https://github.com/code-implementation1/Code7/tree/main/retinanet_resnet152","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[{"task_slug":"object-detection","task_name":"Object Detection"},{"task_slug":"object-detection-1","task_name":"object-detection"}],"methods":[{"method_slug":"1x1-convolution","method_name":"1x1 Convolution"},{"method_slug":"average-pooling","method_name":"Average Pooling"},{"method_slug":"batch-normalization","method_name":"Batch Normalization"},{"method_slug":"bottleneck-residual-block","method_name":"Bottleneck Residual Block"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"fpn","method_name":"FPN"},{"method_slug":"focal-loss","method_name":"Focal Loss"},{"method_slug":"global-average-pooling","method_name":"Global Average Pooling"},{"method_slug":"kaiming-initialization","method_name":"Kaiming Initialization"},{"method_slug":"max-pooling","method_name":"Max Pooling"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-block","method_name":"Residual Block"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"retinanet","method_name":"RetinaNet"},{"method_slug":"sasa","method_name":"SASA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1906.05909","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.05909"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/google-research","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/code-implementation1/Code7/tree/main/retinanet_resnet152","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/The-AI-Summer/self_attention","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/JoeRoussy/adaptive-attention-in-cv","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MaheepChaudhary/Stand-Alone_Self-Attention","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Doyosae/Stand-Alone-Self-Attention","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leaderj1001/Stand-Alone-Self-Attention","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MartinGer/Stand-Alone-Self-Attention-in-Vision-Models","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"listed":{"samples":3,"ran":1,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"62d701bd9f6f7ffd","entry":"get_logger","repo":"JoeRoussy/adaptive-attention-in-cv","repo_kind":"listed","path":"config.py","file_url":"https://github.com/JoeRoussy/adaptive-attention-in-cv/blob/HEAD/config.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"62d701bd9f6f7ffd"}},{"code_sha256_prefix":"5e6067e23503825f","entry":"load_data","repo":"leaderj1001/Stand-Alone-Self-Attention","repo_kind":"listed","path":"preprocess.py","file_url":"https://github.com/leaderj1001/Stand-Alone-Self-Attention/blob/HEAD/preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e6067e23503825f"}},{"code_sha256_prefix":"10b86d2ef7ff73c5","entry":"prepare_data","repo":"MartinGer/Stand-Alone-Self-Attention-in-Vision-Models","repo_kind":"listed","path":"train_SASA.py","file_url":"https://github.com/MartinGer/Stand-Alone-Self-Attention-in-Vision-Models/blob/HEAD/train_SASA.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10b86d2ef7ff73c5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}