{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/accelerating-transformers-with-spectrum-1","title":"Accelerating Transformers with Spectrum-Preserving Token Merging","arxiv_id":"2405.16148","date":"2024-05-25","proceeding":null,"authors":["Hoai-Chau Tran","Duy M. H. Nguyen","Duy M. Nguyen","Trung-Tin Nguyen","Ngan Le","Pengtao Xie","Daniel Sonntag","James Y. Zou","Binh T. Nguyen","Mathias Niepert"],"abstract":"Increasing the throughput of the Transformer architecture, a foundational component used in numerous state-of-the-art models for vision and language tasks (e.g., GPT, LLaVa), is an important problem in machine learning. One recent and effective strategy is to merge token representations within Transformer models, aiming to reduce computational and memory requirements while maintaining accuracy. Prior works have proposed algorithms based on Bipartite Soft Matching (BSM), which divides tokens into distinct sets and merges the top k similar tokens. However, these methods have significant drawbacks, such as sensitivity to token-splitting strategies and damage to informative tokens in later layers. This paper presents a novel paradigm called PiToMe, which prioritizes the preservation of informative tokens using an additional metric termed the energy score. This score identifies large clusters of similar tokens as high-energy, indicating potential candidates for merging, while smaller (unique and isolated) clusters are considered as low-energy and preserved. Experimental findings demonstrate that PiToMe saved from 40-60\\% FLOPs of the base models while exhibiting superior off-the-shelf performance on image classification (0.5\\% average performance drop of ViT-MAE-H compared to 2.6\\% as baselines), image-text retrieval (0.3\\% average performance drop of CLIP on Flickr30k compared to 4.5\\% as others), and analogously in visual questions answering with LLaVa-7B. Furthermore, PiToMe is theoretically shown to preserve intrinsic spectral properties of the original token space under mild conditions","url_abs":"https://arxiv.org/abs/2405.16148v2","url_pdf":"https://arxiv.org/pdf/2405.16148v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"accelerating-transformers-with-spectrum-1","repo_url":"https://github.com/hchautran/PiToMe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-text-retrieval","task_name":"Image-text Retrieval"},{"task_slug":"text-retrieval","task_name":"Text Retrieval"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"base","method_name":"BASE"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.16148","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16148"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hchautran/PiToMe","reach":null}],"summary":{"ran_fixture":4,"ran_honours":2,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"4c11239b70cdc49f","entry":"bsm","repo":"hchautran/PiToMe","repo_kind":"official","path":"algo/pitome/merge.py","file_url":"https://github.com/hchautran/PiToMe/blob/HEAD/algo/pitome/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4c11239b70cdc49f"}},{"code_sha256_prefix":"5d873f7e40563e31","entry":"calculate_cross_flops","repo":"hchautran/PiToMe","repo_kind":"official","path":"main_itr.py","file_url":"https://github.com/hchautran/PiToMe/blob/HEAD/main_itr.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5d873f7e40563e31"}},{"code_sha256_prefix":"6543fa3e4bfa285f","entry":"do_nothing","repo":"hchautran/PiToMe","repo_kind":"official","path":"algo/pitome/merge.py","file_url":"https://github.com/hchautran/PiToMe/blob/HEAD/algo/pitome/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"6543fa3e4bfa285f"}},{"code_sha256_prefix":"3231510f96e375d9","entry":"get_gflops","repo":"hchautran/PiToMe","repo_kind":"official","path":"main_itr.py","file_url":"https://github.com/hchautran/PiToMe/blob/HEAD/main_itr.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3231510f96e375d9"}},{"code_sha256_prefix":"20d1c468c053d9be","entry":"pitome","repo":"hchautran/PiToMe","repo_kind":"official","path":"algo/pitome/merge.py","file_url":"https://github.com/hchautran/PiToMe/blob/HEAD/algo/pitome/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"20d1c468c053d9be"}},{"code_sha256_prefix":"75bb9bdb28dd163f","entry":"pitome","repo":"hchautran/PiToMe","repo_kind":"official","path":"algo/pitome/merge.py","file_url":"https://github.com/hchautran/PiToMe/blob/HEAD/algo/pitome/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"75bb9bdb28dd163f"}},{"code_sha256_prefix":"3d5861107eab1eed","entry":"process_image","repo":"hchautran/PiToMe","repo_kind":"official","path":"main_ic.py","file_url":"https://github.com/hchautran/PiToMe/blob/HEAD/main_ic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3d5861107eab1eed"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}