{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/streamline-without-sacrifice-squeeze-out","title":"Streamline Without Sacrifice -- Squeeze out Computation Redundancy in LMM","arxiv_id":"2505.15816","date":"2025-05-21","proceeding":null,"authors":["Penghao Wu","Lewei Lu","Ziwei Liu"],"abstract":"Large multimodal models excel in multimodal tasks but face significant computational challenges due to excessive computation on visual tokens. Unlike token reduction methods that focus on token-level redundancy, we identify and study the computation-level redundancy on vision tokens to ensure no information loss. Our key insight is that vision tokens from the pretrained vision encoder do not necessarily require all the heavy operations (e.g., self-attention, FFNs) in decoder-only LMMs and could be processed more lightly with proper designs. We designed a series of experiments to discover and progressively squeeze out the vision-related computation redundancy. Based on our findings, we propose ProxyV, a novel approach that utilizes proxy vision tokens to alleviate the computational burden on original vision tokens. ProxyV enhances efficiency without compromising performance and can even yield notable performance gains in scenarios with more moderate efficiency improvements. Furthermore, the flexibility of ProxyV is demonstrated through its combination with token reduction methods to boost efficiency further. The code will be made public at this https://github.com/penghao-wu/ProxyV URL.","url_abs":"https://arxiv.org/abs/2505.15816v1","url_pdf":"https://arxiv.org/pdf/2505.15816v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"streamline-without-sacrifice-squeeze-out","repo_url":"https://github.com/penghao-wu/proxyv","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"token-reduction","task_name":"Token Reduction"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"url","method_name":"URL"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2505.15816","atlas_url":"https://app.syntology.ai/?focus=2505.15816","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.15816"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/penghao-wu/proxyv","reach":null}],"summary":{"ran_draft_wrong":4,"ran_violates":1,"unverified":4},"by_repo_kind":{"official":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"4d75570b4e91d0b4","entry":"FeedForward","repo":"penghao-wu/proxyv","repo_kind":"official","path":"llava/model/multimodal_resampler/perceiver.py","file_url":"https://github.com/penghao-wu/proxyv/blob/HEAD/llava/model/multimodal_resampler/perceiver.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d75570b4e91d0b4"}},{"code_sha256_prefix":"0a65604f002e6ce4","entry":"convert_to_custom_text_state_dict","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"0a65604f002e6ce4"}},{"code_sha256_prefix":"7bff6f3b3cb560e7","entry":"convert_to_custom_text_state_dict","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"7bff6f3b3cb560e7"}},{"code_sha256_prefix":"61d9ad9efd32efee","entry":"get_cast_dtype","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"61d9ad9efd32efee"}},{"code_sha256_prefix":"dcd422d66b0581d8","entry":"get_cast_dtype","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"dcd422d66b0581d8"}},{"code_sha256_prefix":"6a96fbd946c4017a","entry":"PerceiverAttention","repo":"penghao-wu/proxyv","repo_kind":"official","path":"llava/model/multimodal_resampler/perceiver.py","file_url":"https://github.com/penghao-wu/proxyv/blob/HEAD/llava/model/multimodal_resampler/perceiver.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6a96fbd946c4017a"}},{"code_sha256_prefix":"98b4bae0261b6b65","entry":"PerceiverResampler","repo":"penghao-wu/proxyv","repo_kind":"official","path":"llava/model/multimodal_resampler/perceiver.py","file_url":"https://github.com/penghao-wu/proxyv/blob/HEAD/llava/model/multimodal_resampler/perceiver.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"98b4bae0261b6b65"}},{"code_sha256_prefix":"930ebfa80c5b9f00","entry":"PerceiverResamplerModule","repo":"penghao-wu/proxyv","repo_kind":"official","path":"llava/model/multimodal_resampler/perceiver.py","file_url":"https://github.com/penghao-wu/proxyv/blob/HEAD/llava/model/multimodal_resampler/perceiver.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"930ebfa80c5b9f00"}},{"code_sha256_prefix":"a8e9981d9caacade","entry":"build_model_from_openai_state_dict","repo":"penghao-wu/proxyv","repo_kind":"official","path":"llava/model/multimodal_encoder/dev_eva_clip/eva_clip/model.py","file_url":"https://github.com/penghao-wu/proxyv/blob/HEAD/llava/model/multimodal_encoder/dev_eva_clip/eva_clip/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a8e9981d9caacade"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}