{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/not-all-patches-are-what-you-need-expediting","title":"Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations","arxiv_id":"2202.07800","date":"2022-02-16","proceeding":null,"authors":["Youwei Liang","Chongjian Ge","Zhan Tong","Yibing Song","Jue Wang","Pengtao Xie"],"abstract":"Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them. Complete leverage of these image tokens brings redundant computations since not all the tokens are attentive in MHSA. Examples include that tokens containing semantically meaningless or distractive image backgrounds do not positively contribute to the ViT predictions. In this work, we propose to reorganize image tokens during the feed-forward process of ViT models, which is integrated into ViT during training. For each forward inference, we identify the attentive image tokens between MHSA and FFN (i.e., feed-forward network) modules, which is guided by the corresponding class token attention. Then, we reorganize image tokens by preserving attentive image tokens and fusing inattentive ones to expedite subsequent MHSA and FFN computations. To this end, our method EViT improves ViTs from two perspectives. First, under the same amount of input image tokens, our method reduces MHSA and FFN computation for efficient inference. For instance, the inference speed of DeiT-S is increased by 50% while its recognition accuracy is decreased by only 0.3% for ImageNet classification. Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution images. An example is that we improve the recognition accuracy of DeiT-S by 1% for ImageNet classification at the same computational cost of a vanilla DeiT-S. Meanwhile, our method does not introduce more parameters to ViTs. Experiments on the standard benchmarks show the effectiveness of our method. The code is available at https://github.com/youweiliang/evit","url_abs":"https://arxiv.org/abs/2202.07800v2","url_pdf":"https://arxiv.org/pdf/2202.07800v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"not-all-patches-are-what-you-need-expediting","repo_url":"https://github.com/youweiliang/evit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"not-all-patches-are-what-you-need-expediting","repo_url":"https://github.com/shiming-chen/zslvit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"efficient-vits","task_name":"Efficient ViTs"}],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-lv-vit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (With LV-ViT-S)","model":"EViT (70%)","rank_in_archive_order":10,"of":19,"metrics":{"GFLOPs":"4.7","Top 1 Accuracy":"83.0"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-lv-vit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (With LV-ViT-S)","model":"EViT (50%)","rank_in_archive_order":16,"of":19,"metrics":{"GFLOPs":"3.9","Top 1 Accuracy":"82.5"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-deit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (with DeiT-S)","model":"EViT (80%)","rank_in_archive_order":7,"of":41,"metrics":{"GFLOPs":"3.5","Top 1 Accuracy":"79.8"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-deit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (with DeiT-S)","model":"EViT (90%)","rank_in_archive_order":10,"of":41,"metrics":{"GFLOPs":"4.0","Top 1 Accuracy":"79.8"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-deit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (with DeiT-S)","model":"EViT (70%)","rank_in_archive_order":21,"of":41,"metrics":{"GFLOPs":"3.0","Top 1 Accuracy":"79.5"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-deit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (with DeiT-S)","model":"EViT (60%)","rank_in_archive_order":32,"of":41,"metrics":{"GFLOPs":"2.6","Top 1 Accuracy":"78.9"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-deit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (with DeiT-S)","model":"EViT (50%)","rank_in_archive_order":37,"of":41,"metrics":{"GFLOPs":"2.3","Top 1 Accuracy":"78.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.07800","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.07800"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/youweiliang/evit","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shiming-chen/zslvit","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_fixture":2,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"official":{"samples":12,"ran":4,"repositories":1},"community":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"66733f50aff69220","entry":"adjust_keep_rate","repo":"youweiliang/evit","repo_kind":"official","path":"helpers.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"66733f50aff69220"}},{"code_sha256_prefix":"39eace7e2822504f","entry":"drop_path","repo":"youweiliang/evit","repo_kind":"official","path":"evit.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/evit.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"39eace7e2822504f"}},{"code_sha256_prefix":"5ce6e78745861c0e","entry":"merge","repo":"Retinal-Research/EVIT-UNET","repo_kind":"community","path":"unet/eff_unet.py","file_url":"https://github.com/Retinal-Research/EVIT-UNET/blob/HEAD/unet/eff_unet.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ce6e78745861c0e"}},{"code_sha256_prefix":"191b2735955d18f4","entry":"resize_pos_embed","repo":"youweiliang/evit","repo_kind":"official","path":"evit.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/evit.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"191b2735955d18f4"}},{"code_sha256_prefix":"e294b62d47bcc26f","entry":"speed_test","repo":"youweiliang/evit","repo_kind":"official","path":"helpers.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"e294b62d47bcc26f"}},{"code_sha256_prefix":"560e3160f50add18","entry":"stem","repo":"Retinal-Research/EVIT-UNET","repo_kind":"community","path":"unet/eff_unet.py","file_url":"https://github.com/Retinal-Research/EVIT-UNET/blob/HEAD/unet/eff_unet.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"560e3160f50add18"}},{"code_sha256_prefix":"3bc74137ab36aa79","entry":"build_transform","repo":"youweiliang/evit","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"3bc74137ab36aa79"}},{"code_sha256_prefix":"6d8a1f8a0697180f","entry":"checkpoint_filter_fn","repo":"youweiliang/evit","repo_kind":"official","path":"evit.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/evit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"6d8a1f8a0697180f"}},{"code_sha256_prefix":"9e7bf674714e06d5","entry":"dumps_pyarrow","repo":"youweiliang/evit","repo_kind":"official","path":"folder2lmdb.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/folder2lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"9e7bf674714e06d5"}},{"code_sha256_prefix":"efdf8b9d60927bcd","entry":"get_deeper_idx","repo":"youweiliang/evit","repo_kind":"official","path":"visualize_mask.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/visualize_mask.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"efdf8b9d60927bcd"}},{"code_sha256_prefix":"22f1fefd3b89cf1f","entry":"get_real_idx","repo":"youweiliang/evit","repo_kind":"official","path":"visualize_mask.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/visualize_mask.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"22f1fefd3b89cf1f"}},{"code_sha256_prefix":"4ce559065d5b1b96","entry":"load_from","repo":"Retinal-Research/EVIT-UNET","repo_kind":"community","path":"train_unetKD.py","file_url":"https://github.com/Retinal-Research/EVIT-UNET/blob/HEAD/train_unetKD.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ce559065d5b1b96"}},{"code_sha256_prefix":"b4c154b6f0568b6b","entry":"loads_pyarrow","repo":"youweiliang/evit","repo_kind":"official","path":"folder2lmdb.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/folder2lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"b4c154b6f0568b6b"}},{"code_sha256_prefix":"5112560529a3d643","entry":"mask","repo":"youweiliang/evit","repo_kind":"official","path":"visualize_mask.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/visualize_mask.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"5112560529a3d643"}},{"code_sha256_prefix":"51738010a21f76a1","entry":"raw_reader","repo":"youweiliang/evit","repo_kind":"official","path":"folder2lmdb.py","file_url":"https://github.com/youweiliang/evit/blob/HEAD/folder2lmdb.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":false,"mcp_get_code":{"code_sha256":"51738010a21f76a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}