{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/hybrid-level-instruction-injection-for-video","title":"Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models","arxiv_id":"2503.16036","date":"2025-03-20","proceeding":"CVPR 2025 1","authors":["Zhihang Liu","Chen-Wei Xie","Pandeng Li","Liming Zhao","Longxiang Tang","Yun Zheng","Chuanbin Liu","Hongtao Xie"],"abstract":"Recent Multi-modal Large Language Models (MLLMs) have been challenged by the computational overhead resulting from massive video frames, often alleviated through compression strategies. However, the visual content is not equally contributed to user instructions, existing strategies (\\eg, average pool) inevitably lead to the loss of potentially useful information. To tackle this, we propose the Hybrid-level Instruction Injection Strategy for Conditional Token Compression in MLLMs (HICom), utilizing the instruction as a condition to guide the compression from both local and global levels. This encourages the compression to retain the maximum amount of user-focused information while reducing visual tokens to minimize computational burden. Specifically, the instruction condition is injected into the grouped visual tokens at the local level and the learnable tokens at the global level, and we conduct the attention mechanism to complete the conditional compression. From the hybrid-level compression, the instruction-relevant visual parts are highlighted while the temporal-spatial structure is also preserved for easier understanding of LLMs. To further unleash the potential of HICom, we introduce a new conditional pre-training stage with our proposed dataset HICom-248K. Experiments show that our HICom can obtain distinguished video understanding ability with fewer tokens, increasing the performance by 2.43\\% average on three multiple-choice QA benchmarks and saving 78.8\\% tokens compared with the SOTA method. The code is available at https://github.com/lntzm/HICom.","url_abs":"https://arxiv.org/abs/2503.16036v1","url_pdf":"https://arxiv.org/pdf/2503.16036v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"hybrid-level-instruction-injection-for-video","repo_url":"https://github.com/lntzm/hicom","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2503.16036","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.16036"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lntzm/hicom","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/lntzm/HICom","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_fixture":1},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"640230d27ec3a711","entry":"HIComProjector","repo":"lntzm/HICom","repo_kind":"official","path":"hicom/model/projector.py","file_url":"https://github.com/lntzm/HICom/blob/HEAD/hicom/model/projector.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"640230d27ec3a711"}},{"code_sha256_prefix":"fef413d047134d7a","entry":"post_process_visual_feature","repo":"lntzm/HICom","repo_kind":"official","path":"hicom/model/projector.py","file_url":"https://github.com/lntzm/HICom/blob/HEAD/hicom/model/projector.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fef413d047134d7a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}