{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/keyframe-oriented-vision-token-pruning","title":"Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processing","arxiv_id":"2503.10742","date":"2025-03-13","proceeding":null,"authors":["Yudong Liu","Jingwei Sun","Yueqian Lin","Jingyang Zhang","Ming Yin","Qinsi Wang","Jianyi Zhang","Hai Li","Yiran Chen"],"abstract":"Vision language models (VLMs) demonstrate strong capabilities in jointly processing visual and textual data. However, they often incur substantial computational overhead due to redundant visual information, particularly in long-form video scenarios. Existing approaches predominantly focus on either vision token pruning, which may overlook spatio-temporal dependencies, or keyframe selection, which identifies informative frames but discards others, thus disrupting contextual continuity. In this work, we propose KVTP (Keyframe-oriented Vision Token Pruning), a novel framework that overcomes the drawbacks of token pruning and keyframe selection. By adaptively assigning pruning rates based on frame relevance to the query, KVTP effectively retains essential contextual information while significantly reducing redundant computation. To thoroughly evaluate the long-form video understanding capacities of VLMs, we curated and reorganized subsets from VideoMME, EgoSchema, and NextQA into a unified benchmark named SparseKV-QA that highlights real-world scenarios with sparse but crucial events. Our experiments with VLMs of various scales show that KVTP can reduce token usage by 80% without compromising spatiotemporal and contextual consistency, significantly cutting computation while maintaining the performance. These results demonstrate our approach's effectiveness in efficient long-video processing, facilitating more scalable VLM deployment.","url_abs":"https://arxiv.org/abs/2503.10742v1","url_pdf":"https://arxiv.org/pdf/2503.10742v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"keyframe-oriented-vision-token-pruning","repo_url":"https://github.com/1999Lyd/KVTP","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"EgoSchema"},{"task_slug":"form","task_name":"Form"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"focus","method_name":"Focus"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.10742","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.10742"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/1999Lyd/KVTP","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":4,"ran_fixture":2,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e3bf22708b14ee58","entry":"complement_idx","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/model/llava_arch.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/model/llava_arch.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3bf22708b14ee58"}},{"code_sha256_prefix":"b0a851cae92754ff","entry":"extract_patches","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0a851cae92754ff"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"bce1bfc997cf14f9","entry":"outlier_dectection","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/model/llava_arch.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/model/llava_arch.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bce1bfc997cf14f9"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/eval/model_vqa.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/eval/model_vqa.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"7606525af238fb64","entry":"unpad_image","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/model/llava_arch.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/model/llava_arch.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7606525af238fb64"}},{"code_sha256_prefix":"3c8823948724e269","entry":"auto_pad_images","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c8823948724e269"}},{"code_sha256_prefix":"f89aea892269a576","entry":"load_json","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/combine.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/combine.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f89aea892269a576"}},{"code_sha256_prefix":"c80358dff96c25ce","entry":"resize_and_center_crop","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/mm_utils.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c80358dff96c25ce"}},{"code_sha256_prefix":"b20705b0dfd77ffc","entry":"safe_load_tokenizer","repo":"1999Lyd/KVTP","repo_kind":"official","path":"llava/conversation.py","file_url":"https://github.com/1999Lyd/KVTP/blob/HEAD/llava/conversation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b20705b0dfd77ffc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}