{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/trace-temporal-grounding-video-llm-via-causal","title":"TRACE: Temporal Grounding Video LLM via Causal Event Modeling","arxiv_id":"2410.05643","date":"2024-10-08","proceeding":null,"authors":["Yongxin Guo","Jingyu Liu","Mingda Li","Xiaoying Tang","Qingbin Liu","Xi Chen"],"abstract":"Video Temporal Grounding (VTG) is a crucial capability for video understanding models and plays a vital role in downstream tasks such as video browsing and editing. To effectively handle various tasks simultaneously and enable zero-shot prediction, there is a growing trend in employing video LLMs for VTG tasks. However, current video LLM-based methods rely exclusively on natural language generation, lacking the ability to model the clear structure inherent in videos, which restricts their effectiveness in tackling VTG tasks. To address this issue, this paper first formally introduces causal event modeling framework, which represents videos as sequences of events, and predict the current event using previous events, video inputs, and textural instructions. Each event consists of three components: timestamps, salient scores, and textual captions. We then propose a novel task-interleaved video LLM called TRACE to effectively implement the causal event modeling framework in practice. The TRACE processes visual frames, timestamps, salient scores, and text as distinct tasks, employing various encoders and decoding heads for each. Task tokens are arranged in an interleaved sequence according to the causal event modeling framework's formulation. Extensive experiments on various VTG tasks and datasets demonstrate the superior performance of TRACE compared to state-of-the-art video LLMs. Our model and code are available at \\url{https://github.com/gyxxyg/TRACE}.","url_abs":"https://arxiv.org/abs/2410.05643v2","url_pdf":"https://arxiv.org/pdf/2410.05643v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"trace-temporal-grounding-video-llm-via-causal","repo_url":"https://github.com/gyxxyg/trace","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"text-generation","task_name":"Text Generation"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.05643","atlas_url":"https://app.syntology.ai/?focus=2410.05643","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05643"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/gyxxyg/TRACE","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gyxxyg/trace","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"ran_fixture":1,"unverified":10},"by_repo_kind":{"official":{"samples":16,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"741e5b0b3aa32a32","entry":"load_data","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/eval/evaluate.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/eval/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"741e5b0b3aa32a32"}},{"code_sha256_prefix":"4b339b70f2a3e67e","entry":"merge_scenes","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/mm_utils.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/mm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b339b70f2a3e67e"}},{"code_sha256_prefix":"1e366edf3b7d93ae","entry":"merge_scenes_not_exeed_max_scene_num","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/mm_utils.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/mm_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e366edf3b7d93ae"}},{"code_sha256_prefix":"d650b26a59df9e29","entry":"merge_seg_caps","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/eval/evaluate.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/eval/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d650b26a59df9e29"}},{"code_sha256_prefix":"e49f0cf401af904b","entry":"read_txt","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/eval/evaluate.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/eval/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e49f0cf401af904b"}},{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/trace_trainer.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/trace_trainer.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"fe9f9161144b6f90","entry":"TraceMetaForCausalLM","repo":"gyxxyg/trace","repo_kind":"official","path":"trace/model/trace_arch.py","file_url":"https://github.com/gyxxyg/trace/blob/HEAD/trace/model/trace_arch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe9f9161144b6f90"}},{"code_sha256_prefix":"8dc94b9682ebebf9","entry":"build_logger","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/utils.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8dc94b9682ebebf9"}},{"code_sha256_prefix":"bb35e3ac741bb2c9","entry":"get_mm_adapter_state_maybe_zero_3","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/trace_trainer.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/trace_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb35e3ac741bb2c9"}},{"code_sha256_prefix":"fa1225dfac92bc0d","entry":"get_peft_state_maybe_zero_3","repo":"gyxxyg/trace","repo_kind":"official","path":"trace/train_mt.py","file_url":"https://github.com/gyxxyg/trace/blob/HEAD/trace/train_mt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fa1225dfac92bc0d"}},{"code_sha256_prefix":"1c53657305b66e9f","entry":"get_peft_state_non_lora_maybe_zero_3","repo":"gyxxyg/trace","repo_kind":"official","path":"trace/train_mt.py","file_url":"https://github.com/gyxxyg/trace/blob/HEAD/trace/train_mt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1c53657305b66e9f"}},{"code_sha256_prefix":"a2c9ee5dff5e2278","entry":"load_pretrained_model","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/model/builder.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/model/builder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a2c9ee5dff5e2278"}},{"code_sha256_prefix":"735025744c1ab0cf","entry":"maybe_zero_3","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/trace_trainer.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/trace_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"735025744c1ab0cf"}},{"code_sha256_prefix":"616ffbdc154ed2d8","entry":"maybe_zero_3","repo":"gyxxyg/trace","repo_kind":"official","path":"trace/train_mt.py","file_url":"https://github.com/gyxxyg/trace/blob/HEAD/trace/train_mt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"616ffbdc154ed2d8"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/utils.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"gyxxyg/TRACE","repo_kind":"official","path":"trace/utils.py","file_url":"https://github.com/gyxxyg/TRACE/blob/HEAD/trace/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}