{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/framefusion-combining-similarity-and","title":"FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Visual Language Models","arxiv_id":"2501.01986","date":"2024-12-30","proceeding":null,"authors":["Tianyu Fu","Tengxuan Liu","Qinghao Han","Guohao Dai","Shengen Yan","Huazhong Yang","Xuefei Ning","Yu Wang"],"abstract":"The increasing demand to process long and high-resolution videos significantly burdens Large Vision-Language Models (LVLMs) due to the enormous number of visual tokens. Existing token reduction methods primarily focus on importance-based token pruning, which overlooks the redundancy caused by frame resemblance and repetitive visual elements. In this paper, we analyze the high vision token similarities in LVLMs. We reveal that token similarity distribution condenses as layers deepen while maintaining ranking consistency. Leveraging the unique properties of similarity over importance, we introduce FrameFusion, a novel approach that combines similarity-based merging with importance-based pruning for better token reduction in LVLMs. FrameFusion identifies and merges similar tokens before pruning, opening up a new perspective for token reduction. We evaluate FrameFusion on diverse LVLMs, including Llava-Video-{7B,32B,72B}, and MiniCPM-V-8B, on video understanding, question-answering, and retrieval benchmarks. Experiments show that FrameFusion reduces vision tokens by 70$\\%$, achieving 3.4-4.4x LLM speedups and 1.6-1.9x end-to-end speedups, with an average performance impact of less than 3$\\%$. Our code is available at https://github.com/thu-nics/FrameFusion.","url_abs":"https://arxiv.org/abs/2501.01986v1","url_pdf":"https://arxiv.org/pdf/2501.01986v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"framefusion-combining-similarity-and","repo_url":"https://github.com/thu-nics/framefusion","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"token-reduction","task_name":"Token Reduction"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2501.01986","atlas_url":"https://app.syntology.ai/?focus=2501.01986","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.01986"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-nics/framefusion","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran":8,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/internvl/modeling_internlm2.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/internvl/modeling_internlm2.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"12a855af63d45bc2","entry":"Qwen2DecoderLayer_merge_then_prune_by_cost_forward","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/qwen2/modeling_qwen2.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/qwen2/modeling_qwen2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"12a855af63d45bc2"}},{"code_sha256_prefix":"682eafc6f1711f42","entry":"Qwen2VLDecoderLayer_fastv_forward","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/qwen2/modeling_qwen2_vl_baseline.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/qwen2/modeling_qwen2_vl_baseline.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"682eafc6f1711f42"}},{"code_sha256_prefix":"3153f92d89ff16c1","entry":"Qwen2VLDecoderLayer_merge_then_fastv_cost_given_forward","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/qwen2/modeling_qwen2_vl.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/qwen2/modeling_qwen2_vl.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3153f92d89ff16c1"}},{"code_sha256_prefix":"5661336a3d308111","entry":"Qwen2VLModel_merge_then_fastv_cost_given_forward","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/qwen2/modeling_qwen2_vl.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/qwen2/modeling_qwen2_vl.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5661336a3d308111"}},{"code_sha256_prefix":"264c5b9a2f25be7e","entry":"compute_density_overhead","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/qwen2/modeling_qwen2_baseline.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/qwen2/modeling_qwen2_baseline.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"264c5b9a2f25be7e"}},{"code_sha256_prefix":"ccac3bd0c453b847","entry":"get_attr_by_name","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/utils.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ccac3bd0c453b847"}},{"code_sha256_prefix":"6f3669639f7ffa02","entry":"get_vllm_embedding","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/minicpmv/modeling_minicpmv.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/minicpmv/modeling_minicpmv.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f3669639f7ffa02"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/internvl/modeling_internlm2.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/internvl/modeling_internlm2.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"2e002934a51f4e51","entry":"scaled_dot_product_attention","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/utils.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e002934a51f4e51"}},{"code_sha256_prefix":"18116389c0c9cc0c","entry":"Qwen2VLModel_fastv_forward","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/qwen2/modeling_qwen2_vl_baseline.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/qwen2/modeling_qwen2_vl_baseline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18116389c0c9cc0c"}},{"code_sha256_prefix":"c028aebfc37921e4","entry":"apply_rotary_pos_emb","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/internvl/modeling_internlm2.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/internvl/modeling_internlm2.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c028aebfc37921e4"}},{"code_sha256_prefix":"421deb414feba11c","entry":"generate","repo":"thu-nics/framefusion","repo_kind":"official","path":"framefusion/models/internvl/modeling_internvl_chat.py","file_url":"https://github.com/thu-nics/framefusion/blob/HEAD/framefusion/models/internvl/modeling_internvl_chat.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"421deb414feba11c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}