{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ovo-bench-how-far-is-your-video-llms-from","title":"OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?","arxiv_id":"2501.05510","date":"2025-01-09","proceeding":"CVPR 2025 1","authors":["Yifei Li","Junbo Niu","Ziyang Miao","Chunjiang Ge","Yuanhang Zhou","Qihao He","Xiaoyi Dong","Haodong Duan","Shuangrui Ding","Rui Qian","Pan Zhang","Yuhang Zang","Yuhang Cao","Conghui He","Jiaqi Wang"],"abstract":"Temporal Awareness, the ability to reason dynamically based on the timestamp when a question is raised, is the key distinction between offline and online video LLMs. Unlike offline models, which rely on complete videos for static, post hoc analysis, online models process video streams incrementally and dynamically adapt their responses based on the timestamp at which the question is posed. Despite its significance, temporal awareness has not been adequately evaluated in existing benchmarks. To fill this gap, we present OVO-Bench (Online-VideO-Benchmark), a novel video benchmark that emphasizes the importance of timestamps for advanced online video understanding capability benchmarking. OVO-Bench evaluates the ability of video LLMs to reason and respond to events occurring at specific timestamps under three distinct scenarios: (1) Backward tracing: trace back to past events to answer the question. (2) Real-time understanding: understand and respond to events as they unfold at the current timestamp. (3) Forward active responding: delay the response until sufficient future information becomes available to answer the question accurately. OVO-Bench comprises 12 tasks, featuring 644 unique videos and approximately human-curated 2,800 fine-grained meta-annotations with precise timestamps. We combine automated generation pipelines with human curation. With these high-quality samples, we further developed an evaluation pipeline to systematically query video LLMs along the video timeline. Evaluations of nine Video-LLMs reveal that, despite advancements on traditional benchmarks, current models struggle with online video understanding, showing a significant gap compared to human agents. We hope OVO-Bench will drive progress in video LLMs and inspire future research in online video reasoning. Our benchmark and code can be accessed at https://github.com/JoeLeelyf/OVO-Bench.","url_abs":"https://arxiv.org/abs/2501.05510v2","url_pdf":"https://arxiv.org/pdf/2501.05510v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ovo-bench-how-far-is-your-video-llms-from","repo_url":"https://github.com/joeleelyf/ovo-bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"hoc","method_name":"HOC"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.05510","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.05510"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/joeleelyf/ovo-bench","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/JoeLeelyf/OVO-Bench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_fixture":1,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"57e95166ade5f25c","entry":"build_transform","repo":"JoeLeelyf/OVO-Bench","repo_kind":"official","path":"models/InternVL2.py","file_url":"https://github.com/JoeLeelyf/OVO-Bench/blob/HEAD/models/InternVL2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"57e95166ade5f25c"}},{"code_sha256_prefix":"bd77f5f8067f18e9","entry":"find_closest_aspect_ratio","repo":"JoeLeelyf/OVO-Bench","repo_kind":"official","path":"models/InternVL2.py","file_url":"https://github.com/JoeLeelyf/OVO-Bench/blob/HEAD/models/InternVL2.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd77f5f8067f18e9"}},{"code_sha256_prefix":"eaf0befd020aa0fd","entry":"calculate_diff","repo":"JoeLeelyf/OVO-Bench","repo_kind":"official","path":"models/Dispider.py","file_url":"https://github.com/JoeLeelyf/OVO-Bench/blob/HEAD/models/Dispider.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eaf0befd020aa0fd"}},{"code_sha256_prefix":"fa5f46c5d6bb7316","entry":"get_seq_frames","repo":"JoeLeelyf/OVO-Bench","repo_kind":"official","path":"models/Dispider.py","file_url":"https://github.com/JoeLeelyf/OVO-Bench/blob/HEAD/models/Dispider.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa5f46c5d6bb7316"}},{"code_sha256_prefix":"4f947b95c8631f4b","entry":"get_seq_time","repo":"JoeLeelyf/OVO-Bench","repo_kind":"official","path":"models/Dispider.py","file_url":"https://github.com/JoeLeelyf/OVO-Bench/blob/HEAD/models/Dispider.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f947b95c8631f4b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}