{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/retake-reducing-temporal-and-knowledge","title":"ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding","arxiv_id":"2412.20504","date":"2024-12-29","proceeding":null,"authors":["Xiao Wang","Qingyi Si","Jianlong Wu","Shiyu Zhu","Li Cao","Liqiang Nie"],"abstract":"Video Large Language Models (VideoLLMs) have made significant strides in video understanding but struggle with long videos due to the limitations of their backbone LLMs. Existing solutions rely on length extrapolation, which is memory-constrained, or visual token compression, which primarily leverages low-level temporal redundancy while overlooking the more effective high-level knowledge redundancy. To address this, we propose $\\textbf{ReTaKe}$, a training-free method with two novel modules DPSelect and PivotKV, to jointly reduce both temporal visual redundancy and knowledge redundancy for video compression. To align with the way of human temporal perception, DPSelect identifies keyframes based on inter-frame distance peaks. To leverage LLMs' learned prior knowledge, PivotKV marks the keyframes as pivots and compress non-pivot frames by pruning low-attention tokens in their KV cache. ReTaKe enables VideoLLMs to process 8 times longer frames (up to 2048), outperforming similar-sized models by 3-5% and even rivaling much larger ones on VideoMME, MLVU, LongVideoBench, and LVBench. Moreover, by overlapping compression operations with prefilling, ReTaKe introduces only ~10% prefilling latency overhead while reducing decoding latency by ~20%. Our code is available at https://github.com/SCZwangxiao/video-ReTaKe.","url_abs":"https://arxiv.org/abs/2412.20504v5","url_pdf":"https://arxiv.org/pdf/2412.20504v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"retake-reducing-temporal-and-knowledge","repo_url":"https://github.com/sczwangxiao/video-retake","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"video-compression","task_name":"Video Compression"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"pruning","method_name":"Pruning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.20504","atlas_url":"https://app.syntology.ai/?focus=2412.20504","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.20504"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sczwangxiao/video-retake","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran":10,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":15,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3c76e52815c5401d","entry":"repeat_kv","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/longvideo_cache.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/longvideo_cache.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c76e52815c5401d"}},{"code_sha256_prefix":"81664204c8525de9","entry":"evaluate_mlvu_generation","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/dataset_utils.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81664204c8525de9"}},{"code_sha256_prefix":"d37f4e5bd2ad847b","entry":"load_yaml","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/infer_eval.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/infer_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d37f4e5bd2ad847b"}},{"code_sha256_prefix":"6ad69145ddabafe5","entry":"memory_bank_compress_MALLM","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/visual_compression.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/visual_compression.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ad69145ddabafe5"}},{"code_sha256_prefix":"5f9f5bd51036cfc1","entry":"memory_bank_compress_MALLM_hard","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/visual_compression.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/visual_compression.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f9f5bd51036cfc1"}},{"code_sha256_prefix":"0366d231d1eb184f","entry":"memory_bank_compress_keyframe","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/visual_compression.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/visual_compression.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0366d231d1eb184f"}},{"code_sha256_prefix":"a83c7315b78d5c2c","entry":"patch_llava_onevision_config","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/monkeypatch.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/monkeypatch.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a83c7315b78d5c2c"}},{"code_sha256_prefix":"9e76eeb39428f98d","entry":"patch_qwen2vl_config","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/monkeypatch.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/monkeypatch.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9e76eeb39428f98d"}},{"code_sha256_prefix":"8a68c201ee613f88","entry":"retake_LlavaOnevisionForConditionalGeneration_get_chunk_size","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/llava_onevision.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/llava_onevision.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a68c201ee613f88"}},{"code_sha256_prefix":"7ddc25ea1dac67d2","entry":"retake_LlavaOnevisionForConditionalGeneration_segment_input_ids","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/llava_onevision.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/llava_onevision.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ddc25ea1dac67d2"}},{"code_sha256_prefix":"e03d53ba9d4f9ae5","entry":"rotate_half","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/longvideo_cache.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/longvideo_cache.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e03d53ba9d4f9ae5"}},{"code_sha256_prefix":"2a31a1d556220646","entry":"trimm_results","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/infer_eval.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/infer_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2a31a1d556220646"}},{"code_sha256_prefix":"8971bec8b941516b","entry":"apply_multimodal_rotary_pos_emb","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/longvideo_cache.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/longvideo_cache.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8971bec8b941516b"}},{"code_sha256_prefix":"dd53543386710ddf","entry":"eval_mlvu_results","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/dataset_utils.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dd53543386710ddf"}},{"code_sha256_prefix":"feee0c01f58c3b2e","entry":"eval_videomme_results","repo":"sczwangxiao/video-retake","repo_kind":"official","path":"retake/dataset_utils.py","file_url":"https://github.com/sczwangxiao/video-retake/blob/HEAD/retake/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"feee0c01f58c3b2e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}