{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tc-bench-benchmarking-temporal","title":"TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation","arxiv_id":"2406.08656","date":"2024-06-12","proceeding":null,"authors":["Weixi Feng","Jiachen Li","Michael Saxon","Tsu-Jui Fu","Wenhu Chen","William Yang Wang"],"abstract":"Video generation has many unique challenges beyond those of image generation. The temporal dimension introduces extensive possible variations across frames, over which consistency and continuity may be violated. In this study, we move beyond evaluating simple actions and argue that generated videos should incorporate the emergence of new concepts and their relation transitions like in real-world videos as time progresses. To assess the Temporal Compositionality of video generation models, we propose TC-Bench, a benchmark of meticulously crafted text prompts, corresponding ground truth videos, and robust evaluation metrics. The prompts articulate the initial and final states of scenes, effectively reducing ambiguities for frame development and simplifying the assessment of transition completion. In addition, by collecting aligned real-world videos corresponding to the prompts, we expand TC-Bench's applicability from text-conditional models to image-conditional ones that can perform generative frame interpolation. We also develop new metrics to measure the completeness of component transitions in generated videos, which demonstrate significantly higher correlations with human judgments than existing metrics. Our comprehensive experimental results reveal that most video generators achieve less than 20% of the compositional changes, highlighting enormous space for future improvement. Our analysis indicates that current video generation models struggle to interpret descriptions of compositional changes and synthesize various components across different time steps.","url_abs":"https://arxiv.org/abs/2406.08656v1","url_pdf":"https://arxiv.org/pdf/2406.08656v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tc-bench-benchmarking-temporal","repo_url":"https://github.com/weixi-feng/tc-bench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"image-to-video","task_name":"Image to Video Generation"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.08656","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.08656"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/weixi-feng/tc-bench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5},"by_repo_kind":{"official":{"samples":5,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3d39217a41c77766","entry":"encode_image","repo":"weixi-feng/tc-bench","repo_kind":"official","path":"vlm_eval.py","file_url":"https://github.com/weixi-feng/tc-bench/blob/HEAD/vlm_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3d39217a41c77766"}},{"code_sha256_prefix":"02caab74330abf6b","entry":"encode_image_from_path","repo":"weixi-feng/tc-bench","repo_kind":"official","path":"vlm_eval.py","file_url":"https://github.com/weixi-feng/tc-bench/blob/HEAD/vlm_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02caab74330abf6b"}},{"code_sha256_prefix":"bd3d9e1031c8ee4d","entry":"extract_checks","repo":"weixi-feng/tc-bench","repo_kind":"official","path":"generate_assertions.py","file_url":"https://github.com/weixi-feng/tc-bench/blob/HEAD/generate_assertions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd3d9e1031c8ee4d"}},{"code_sha256_prefix":"8cba1a977e907813","entry":"extract_video_id","repo":"weixi-feng/tc-bench","repo_kind":"official","path":"download_youtube.py","file_url":"https://github.com/weixi-feng/tc-bench/blob/HEAD/download_youtube.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8cba1a977e907813"}},{"code_sha256_prefix":"bc59900741312cc7","entry":"plot_frames","repo":"weixi-feng/tc-bench","repo_kind":"official","path":"vlm_eval.py","file_url":"https://github.com/weixi-feng/tc-bench/blob/HEAD/vlm_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc59900741312cc7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}