{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/joint-token-pruning-and-squeezing-towards","title":"Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers","arxiv_id":"2304.10716","date":"2023-04-21","proceeding":"CVPR 2023 1","authors":["Siyuan Wei","Tianzhu Ye","Shen Zhang","Yao Tang","Jiajun Liang"],"abstract":"Although vision transformers (ViTs) have shown promising results in various computer vision tasks recently, their high computational cost limits their practical applications. Previous approaches that prune redundant tokens have demonstrated a good trade-off between performance and computation costs. Nevertheless, errors caused by pruning strategies can lead to significant information loss. Our quantitative experiments reveal that the impact of pruned tokens on performance should be noticeable. To address this issue, we propose a novel joint Token Pruning & Squeezing module (TPS) for compressing vision transformers with higher efficiency. Firstly, TPS adopts pruning to get the reserved and pruned subsets. Secondly, TPS squeezes the information of pruned tokens into partial reserved tokens via the unidirectional nearest-neighbor matching and similarity-based fusing steps. Compared to state-of-the-art methods, our approach outperforms them under all token pruning intensities. Especially while shrinking DeiT-tiny&small computational budgets to 35%, it improves the accuracy by 1%-6% compared with baselines on ImageNet classification. The proposed method can accelerate the throughput of DeiT-small beyond DeiT-tiny, while its accuracy surpasses DeiT-tiny by 4.78%. Experiments on various transformers demonstrate the effectiveness of our method, while analysis experiments prove our higher robustness to the errors of the token pruning policy. Code is available at https://github.com/megvii-research/TPS-CVPR2023.","url_abs":"https://arxiv.org/abs/2304.10716v1","url_pdf":"https://arxiv.org/pdf/2304.10716v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"joint-token-pruning-and-squeezing-towards","repo_url":"https://github.com/megvii-research/tps-cvpr2023","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"efficient-vits","task_name":"Efficient ViTs"}],"methods":[{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-lv-vit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (With LV-ViT-S)","model":"dTPS","rank_in_archive_order":15,"of":19,"metrics":{"GFLOPs":"3.8","Top 1 Accuracy":"82.6"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-lv-vit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (With LV-ViT-S)","model":"eTPS","rank_in_archive_order":17,"of":19,"metrics":{"GFLOPs":"3.8","Top 1 Accuracy":"82.5"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-deit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (with DeiT-S)","model":"dTPS","rank_in_archive_order":2,"of":41,"metrics":{"GFLOPs":"3.0","Top 1 Accuracy":"80.1"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-deit-s","task":"Efficient ViTs","dataset":"ImageNet-1K (with DeiT-S)","model":"eTPS","rank_in_archive_order":14,"of":41,"metrics":{"GFLOPs":"3.0","Top 1 Accuracy":"79.7"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-deit-t","task":"Efficient ViTs","dataset":"ImageNet-1K (with DeiT-T)","model":"dTPS","rank_in_archive_order":1,"of":22,"metrics":{"GFLOPs":"0.8","Top 1 Accuracy":"72.9"},"uses_additional_data":false},{"leaderboard":"/sota/efficient-vits-on-imagenet-1k-with-deit-t","task":"Efficient ViTs","dataset":"ImageNet-1K (with DeiT-T)","model":"eTPS","rank_in_archive_order":5,"of":22,"metrics":{"GFLOPs":"0.8","Top 1 Accuracy":"72.3"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2304.10716","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2304.10716"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/megvii-research/tps-cvpr2023","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/megvii-research/TPS-CVPR2023","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"908619be92da9903","entry":"batch_index_select","repo":"megvii-research/TPS-CVPR2023","repo_kind":"official","path":"torch_codebase/models/dynamic_vit.py","file_url":"https://github.com/megvii-research/TPS-CVPR2023/blob/HEAD/torch_codebase/models/dynamic_vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"908619be92da9903"}},{"code_sha256_prefix":"3913c620f6926f42","entry":"rand_bbox","repo":"megvii-research/TPS-CVPR2023","repo_kind":"official","path":"megengine_codebase/mixup.py","file_url":"https://github.com/megvii-research/TPS-CVPR2023/blob/HEAD/megengine_codebase/mixup.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3913c620f6926f42"}},{"code_sha256_prefix":"0d5eb6177974e3b2","entry":"build_dataset","repo":"megvii-research/TPS-CVPR2023","repo_kind":"official","path":"torch_codebase/datasets.py","file_url":"https://github.com/megvii-research/TPS-CVPR2023/blob/HEAD/torch_codebase/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0d5eb6177974e3b2"}},{"code_sha256_prefix":"3bc74137ab36aa79","entry":"build_transform","repo":"megvii-research/TPS-CVPR2023","repo_kind":"official","path":"torch_codebase/datasets.py","file_url":"https://github.com/megvii-research/TPS-CVPR2023/blob/HEAD/torch_codebase/datasets.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3bc74137ab36aa79"}},{"code_sha256_prefix":"427f72651ff7cfa0","entry":"contain_bias_and_norm","repo":"megvii-research/TPS-CVPR2023","repo_kind":"official","path":"megengine_codebase/utils.py","file_url":"https://github.com/megvii-research/TPS-CVPR2023/blob/HEAD/megengine_codebase/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"427f72651ff7cfa0"}},{"code_sha256_prefix":"7026a4f946a4f7a3","entry":"get_loss_cls","repo":"megvii-research/TPS-CVPR2023","repo_kind":"official","path":"torch_codebase/losses/loss.py","file_url":"https://github.com/megvii-research/TPS-CVPR2023/blob/HEAD/torch_codebase/losses/loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7026a4f946a4f7a3"}},{"code_sha256_prefix":"c0d55edbcda649c3","entry":"get_module_cls","repo":"megvii-research/TPS-CVPR2023","repo_kind":"official","path":"torch_codebase/models/dynamic_vit.py","file_url":"https://github.com/megvii-research/TPS-CVPR2023/blob/HEAD/torch_codebase/models/dynamic_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c0d55edbcda649c3"}},{"code_sha256_prefix":"b31af37bba5ce554","entry":"get_sim","repo":"megvii-research/TPS-CVPR2023","repo_kind":"official","path":"torch_codebase/models/dynamic_vit.py","file_url":"https://github.com/megvii-research/TPS-CVPR2023/blob/HEAD/torch_codebase/models/dynamic_vit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b31af37bba5ce554"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}