{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/madtp-multimodal-alignment-guided-dynamic","title":"MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer","arxiv_id":"2403.02991","date":"2024-03-05","proceeding":"CVPR 2024 1","authors":["JianJian Cao","Peng Ye","Shengze Li","Chong Yu","Yansong Tang","Jiwen Lu","Tao Chen"],"abstract":"Vision-Language Transformers (VLTs) have shown great success recently, but are meanwhile accompanied by heavy computation costs, where a major reason can be attributed to the large number of visual and language tokens. Existing token pruning research for compressing VLTs mainly follows a single-modality-based scheme yet ignores the critical role of aligning different modalities for guiding the token pruning process, causing the important tokens for one modality to be falsely pruned in another modality branch. Meanwhile, existing VLT pruning works also lack the flexibility to dynamically compress each layer based on different input samples. To this end, we propose a novel framework named Multimodal Alignment-Guided Dynamic Token Pruning (MADTP) for accelerating various VLTs. Specifically, we first introduce a well-designed Multi-modality Alignment Guidance (MAG) module that can align features of the same semantic concept from different modalities, to ensure the pruned tokens are less important for all modalities. We further design a novel Dynamic Token Pruning (DTP) module, which can adaptively adjust the token compression ratio in each layer based on different input instances. Extensive experiments on various benchmarks demonstrate that MADTP significantly reduces the computational complexity of kinds of multimodal models while preserving competitive performance. Notably, when applied to the BLIP model in the NLVR2 dataset, MADTP can reduce the GFLOPs by 80% with less than 4% performance degradation.","url_abs":"https://arxiv.org/abs/2403.02991v1","url_pdf":"https://arxiv.org/pdf/2403.02991v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"madtp-multimodal-alignment-guided-dynamic","repo_url":"https://github.com/double125/madtp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"blip","method_name":"BLIP"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.02991","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.02991"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/double125/madtp","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/double125/MADTP","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"ran_fixture":1,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4452c423028ca1a4","entry":"Query_model","repo":"double125/MADTP","repo_kind":"official","path":"models/utils.py","file_url":"https://github.com/double125/MADTP/blob/HEAD/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4452c423028ca1a4"}},{"code_sha256_prefix":"84a5f885901512fc","entry":"Sparsemax","repo":"double125/MADTP","repo_kind":"official","path":"models/utils.py","file_url":"https://github.com/double125/MADTP/blob/HEAD/models/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"84a5f885901512fc"}},{"code_sha256_prefix":"c7f85279d772ea19","entry":"itm_eval","repo":"double125/madtp","repo_kind":"official","path":"compress_retrieval_clip_dtp.py","file_url":"https://github.com/double125/madtp/blob/HEAD/compress_retrieval_clip_dtp.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7f85279d772ea19"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"double125/madtp","repo_kind":"official","path":"models/blip_retrieval.py","file_url":"https://github.com/double125/madtp/blob/HEAD/models/blip_retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"d1807c35f4bb06d9","entry":"build_model","repo":"double125/madtp","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/double125/madtp/blob/HEAD/clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d1807c35f4bb06d9"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"double125/madtp","repo_kind":"official","path":"clip/model.py","file_url":"https://github.com/double125/madtp/blob/HEAD/clip/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"f9de4cdd8a624801","entry":"rsetattr","repo":"double125/madtp","repo_kind":"official","path":"models/blip_nlvr.py","file_url":"https://github.com/double125/madtp/blob/HEAD/models/blip_nlvr.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9de4cdd8a624801"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}