{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/three-things-everyone-should-know-about","title":"Three things everyone should know about Vision Transformers","arxiv_id":"2203.09795","date":"2022-03-18","proceeding":null,"authors":["Hugo Touvron","Matthieu Cord","Alaaeldin El-Nouby","Jakob Verbeek","Hervé Jégou"],"abstract":"After their initial success in natural language processing, transformer architectures have rapidly gained traction in computer vision, providing state-of-the-art results for tasks such as image classification, detection, segmentation, and video analysis. We offer three insights based on simple and easy to implement variants of vision transformers. (1) The residual layers of vision transformers, which are usually processed sequentially, can to some extent be processed efficiently in parallel without noticeably affecting the accuracy. (2) Fine-tuning the weights of the attention layers is sufficient to adapt vision transformers to a higher resolution and to other classification tasks. This saves compute, reduces the peak memory consumption at fine-tuning time, and allows sharing the majority of weights across tasks. (3) Adding MLP-based patch pre-processing layers improves Bert-like self-supervised training based on patch masking. We evaluate the impact of these design choices using the ImageNet-1k dataset, and confirm our findings on the ImageNet-v2 test set. Transfer performance is measured across six smaller datasets.","url_abs":"https://arxiv.org/abs/2203.09795v1","url_pdf":"https://arxiv.org/pdf/2203.09795v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"three-things-everyone-should-know-about","repo_url":"https://github.com/facebookresearch/deit","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"three-things-everyone-should-know-about","repo_url":"https://github.com/conceptofmind/Parallel-ViT-flax","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"three-things-everyone-should-know-about","repo_url":"https://github.com/lucidrains/vit-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"three-things-everyone-should-know-about","repo_url":"https://github.com/rwightman/pytorch-image-models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"three-things-everyone-should-know-about","repo_url":"https://github.com/code-implementation1/Code6/tree/main/NFNet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok"}},{"paper_slug":"three-things-everyone-should-know-about","repo_url":"https://github.com/code-implementation1/Code9/tree/main/regnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"three-things-everyone-should-know-about","repo_url":"https://github.com/yangyucheng000/University/tree/main/model-3/regnet","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"three-things-everyone-should-know-about","repo_url":"https://gitlab.com/birder/birder","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"fine-grained-image-classification","task_name":"Fine-Grained Image Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"fixres","method_name":"FixRes"},{"method_slug":"layerscale","method_name":"LayerScale"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/fine-grained-image-classification-on-stanford","task":"Fine-Grained Image Classification","dataset":"Stanford Cars","model":"ViT-L (attn finetune)","rank_in_archive_order":62,"of":83,"metrics":{"Accuracy":"93.8%"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"ViT-B (attn fine-tune)","rank_in_archive_order":10,"of":265,"metrics":{"Percentage correct":"99.3"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-cifar-100","task":"Image Classification","dataset":"CIFAR-100","model":"ViT-L (attn fine-tune)","rank_in_archive_order":11,"of":211,"metrics":{"Percentage correct":"93.0"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-flowers-102","task":"Image Classification","dataset":"Flowers-102","model":"ViT-B (attn finetune)","rank_in_archive_order":24,"of":52,"metrics":{"Accuracy":"98.5"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ViT-L@384 (attn finetune)","rank_in_archive_order":218,"of":1060,"metrics":{"Top 1 Accuracy":"85.5%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ViT-B@384 (attn finetune)","rank_in_archive_order":324,"of":1060,"metrics":{"Top 1 Accuracy":"84.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ViT-B-36x1","rank_in_archive_order":347,"of":1060,"metrics":{"Top 1 Accuracy":"84.1%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ViT-B-18x2","rank_in_archive_order":348,"of":1060,"metrics":{"Top 1 Accuracy":"84.1%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ViT-B (hMLP + BeiT)","rank_in_archive_order":425,"of":1060,"metrics":{"Top 1 Accuracy":"83.4%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ViT-S-24x2","rank_in_archive_order":520,"of":1060,"metrics":{"Top 1 Accuracy":"82.6%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet","task":"Image Classification","dataset":"ImageNet","model":"ViT-S-48x1","rank_in_archive_order":549,"of":1060,"metrics":{"Top 1 Accuracy":"82.3%"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-imagenet-v2","task":"Image Classification","dataset":"ImageNet V2","model":"ViT-B-36x1","rank_in_archive_order":19,"of":33,"metrics":{"Top 1 Accuracy":"73.9"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-inaturalist-2018","task":"Image Classification","dataset":"iNaturalist 2018","model":"ViT-L (attn finetune)","rank_in_archive_order":27,"of":60,"metrics":{"Top-1 Accuracy":"75.3%"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.09795","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.09795"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rwightman/pytorch-image-models","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/deit","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lucidrains/vit-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/code-implementation1/Code6/tree/main/NFNet","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/conceptofmind/Parallel-ViT-flax","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://gitlab.com/birder/birder","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangyucheng000/University/tree/main/model-3/regnet","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/code-implementation1/Code9/tree/main/regnet","reach":null}],"summary":{"ran_violates":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6ba8cee9f5daea41","entry":"pair","repo":"conceptofmind/Parallel-ViT-flax","repo_kind":"listed","path":"parallel_vit.py","file_url":"https://github.com/conceptofmind/Parallel-ViT-flax/blob/HEAD/parallel_vit.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ba8cee9f5daea41"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}