{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/timechat-online-80-visual-tokens-are","title":"TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos","arxiv_id":"2504.17343","date":"2025-04-24","proceeding":null,"authors":["Linli Yao","Yicheng Li","Yuancheng Wei","Lei LI","Shuhuai Ren","Yuanxin Liu","Kun Ouyang","Lean Wang","Shicheng Li","Sida Li","Lingpeng Kong","Qi Liu","Yuanxing Zhang","Xu sun"],"abstract":"The rapid growth of online video platforms, particularly live streaming services, has created an urgent need for real-time video understanding systems. These systems must process continuous video streams and respond to user queries instantaneously, presenting unique challenges for current Video Large Language Models (VideoLLMs). While existing VideoLLMs excel at processing complete videos, they face significant limitations in streaming scenarios due to their inability to handle dense, redundant frames efficiently. We introduce TimeChat-Online, a novel online VideoLLM that revolutionizes real-time video interaction. At its core lies our innovative Differential Token Drop (DTD) module, which addresses the fundamental challenge of visual redundancy in streaming videos. Drawing inspiration from human visual perception's Change Blindness phenomenon, DTD preserves meaningful temporal changes while filtering out static, redundant content between frames. Remarkably, our experiments demonstrate that DTD achieves an 82.8% reduction in video tokens while maintaining 98% performance on StreamingBench, revealing that over 80% of visual content in streaming videos is naturally redundant without requiring language guidance. To enable seamless real-time interaction, we present TimeChat-Online-139K, a comprehensive streaming video dataset featuring diverse interaction patterns including backward-tracing, current-perception, and future-responding scenarios. TimeChat-Online's unique Proactive Response capability, naturally achieved through continuous monitoring of video scene transitions via DTD, sets it apart from conventional approaches. Our extensive evaluation demonstrates TimeChat-Online's superior performance on streaming benchmarks (StreamingBench and OvOBench) and maintaining competitive results on long-form video tasks such as Video-MME and MLVU.","url_abs":"https://arxiv.org/abs/2504.17343v1","url_pdf":"https://arxiv.org/pdf/2504.17343v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"timechat-online-80-visual-tokens-are","repo_url":"https://github.com/yaolinli/TimeChat-Online","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"timechat-online-80-visual-tokens-are","repo_url":"https://github.com/renshuhuai-andy/timechat","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"mme","task_name":"MME"},{"task_slug":null,"task_name":"Video MME"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2504.17343","atlas_url":"https://app.syntology.ai/?focus=2504.17343","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.17343"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yaolinli/TimeChat-Online","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/renshuhuai-andy/timechat","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"ran_fixture":2,"ran_violates":1,"ran_draft_wrong":3,"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1},"listed":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"renshuhuai-andy/timechat","repo_kind":"listed","path":"timechat/models/modeling_llama.py","file_url":"https://github.com/renshuhuai-andy/timechat/blob/HEAD/timechat/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"renshuhuai-andy/timechat","repo_kind":"listed","path":"timechat/models/blip2.py","file_url":"https://github.com/renshuhuai-andy/timechat/blob/HEAD/timechat/models/blip2.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"7d214655954e2bc6","entry":"getAttMap","repo":"renshuhuai-andy/timechat","repo_kind":"listed","path":"timechat/common/gradcam.py","file_url":"https://github.com/renshuhuai-andy/timechat/blob/HEAD/timechat/common/gradcam.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"7d214655954e2bc6"}},{"code_sha256_prefix":"98589643273920ba","entry":"main_process","repo":"renshuhuai-andy/timechat","repo_kind":"listed","path":"timechat/common/dist_utils.py","file_url":"https://github.com/renshuhuai-andy/timechat/blob/HEAD/timechat/common/dist_utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"98589643273920ba"}},{"code_sha256_prefix":"791c72070c0b1cfb","entry":"node_to_dict","repo":"renshuhuai-andy/timechat","repo_kind":"listed","path":"timechat/common/config.py","file_url":"https://github.com/renshuhuai-andy/timechat/blob/HEAD/timechat/common/config.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"791c72070c0b1cfb"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"renshuhuai-andy/timechat","repo_kind":"listed","path":"timechat/models/modeling_llama.py","file_url":"https://github.com/renshuhuai-andy/timechat/blob/HEAD/timechat/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"cb33571427334815","entry":"tile","repo":"renshuhuai-andy/timechat","repo_kind":"listed","path":"timechat/models/base_model.py","file_url":"https://github.com/renshuhuai-andy/timechat/blob/HEAD/timechat/models/base_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"cb33571427334815"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"renshuhuai-andy/timechat","repo_kind":"listed","path":"timechat/models/base_model.py","file_url":"https://github.com/renshuhuai-andy/timechat/blob/HEAD/timechat/models/base_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"af2ddebd33bc4882","entry":"concat_two_mp4s","repo":"yaolinli/TimeChat-Online","repo_kind":"official","path":"demo/web_demo.py","file_url":"https://github.com/yaolinli/TimeChat-Online/blob/HEAD/demo/web_demo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"af2ddebd33bc4882"}},{"code_sha256_prefix":"5fb54f24c9f445a4","entry":"convert_webm_to_mp4","repo":"yaolinli/TimeChat-Online","repo_kind":"official","path":"demo/web_demo.py","file_url":"https://github.com/yaolinli/TimeChat-Online/blob/HEAD/demo/web_demo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5fb54f24c9f445a4"}},{"code_sha256_prefix":"643539116e049080","entry":"download_cached_file","repo":"renshuhuai-andy/timechat","repo_kind":"listed","path":"timechat/common/dist_utils.py","file_url":"https://github.com/renshuhuai-andy/timechat/blob/HEAD/timechat/common/dist_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"643539116e049080"}},{"code_sha256_prefix":"6bc45824e4078218","entry":"get_video_duration","repo":"yaolinli/TimeChat-Online","repo_kind":"official","path":"demo/web_demo.py","file_url":"https://github.com/yaolinli/TimeChat-Online/blob/HEAD/demo/web_demo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6bc45824e4078218"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}