{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/t2v-compbench-a-comprehensive-benchmark-for","title":"T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation","arxiv_id":"2407.14505","date":"2024-07-19","proceeding":"CVPR 2025 1","authors":["Kaiyue Sun","Kaiyi Huang","Xian Liu","Yue Wu","Zihan Xu","Zhenguo Li","Xihui Liu"],"abstract":"Text-to-video (T2V) generative models have advanced significantly, yet their ability to compose different objects, attributes, actions, and motions into a video remains unexplored. Previous text-to-video benchmarks also neglect this important ability for evaluation. In this work, we conduct the first systematic study on compositional text-to-video generation. We propose T2V-CompBench, the first benchmark tailored for compositional text-to-video generation. T2V-CompBench encompasses diverse aspects of compositionality, including consistent attribute binding, dynamic attribute binding, spatial relationships, motion binding, action binding, object interactions, and generative numeracy. We further carefully design evaluation metrics of multimodal large language model (MLLM)-based, detection-based, and tracking-based metrics, which can better reflect the compositional text-to-video generation quality of seven proposed categories with 1400 text prompts. The effectiveness of the proposed metrics is verified by correlation with human evaluations. We also benchmark various text-to-video generative models and conduct in-depth analysis across different models and various compositional categories. We find that compositional text-to-video generation is highly challenging for current models, and we hope our attempt could shed light on future research in this direction.","url_abs":"https://arxiv.org/abs/2407.14505v2","url_pdf":"https://arxiv.org/pdf/2407.14505v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"t2v-compbench-a-comprehensive-benchmark-for","repo_url":"https://github.com/KaiyueSun98/T2V-CompBench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"attribute","task_name":"Attribute"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"multimodal-large-language-model","task_name":"Multimodal Large Language Model"},{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.14505","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.14505"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KaiyueSun98/T2V-CompBench","reach":{"status":"ok"}}],"summary":{"ran":5,"ran_honours":2,"unverified":4},"by_repo_kind":{"official":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"18e1081b616ae0e0","entry":"extract_json","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"LLaVA/llava/eval/compbench_eval_action_binding.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/LLaVA/llava/eval/compbench_eval_action_binding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18e1081b616ae0e0"}},{"code_sha256_prefix":"0b9124dcf2830cbb","entry":"extract_json","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"LLaVA/llava/eval/compbench_eval_consistent_attr.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/LLaVA/llava/eval/compbench_eval_consistent_attr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0b9124dcf2830cbb"}},{"code_sha256_prefix":"fcf9191e44298633","entry":"get_rainbow_colors","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"dot/compbench_eval_motion_binding.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/dot/compbench_eval_motion_binding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fcf9191e44298633"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"LLaVA/llava/eval/compbench_eval_dynamic_attr.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/LLaVA/llava/eval/compbench_eval_dynamic_attr.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"d5044bdde33c54c2","entry":"load_images","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"LLaVA/llava/eval/compbench_eval_dynamic_attr.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/LLaVA/llava/eval/compbench_eval_dynamic_attr.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d5044bdde33c54c2"}},{"code_sha256_prefix":"4e1466e9857ababe","entry":"plot_boxes_to_image","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"Grounded-Segment-Anything/GroundingDINO/demo/compbench_eval_numeracy.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/Grounded-Segment-Anything/GroundingDINO/demo/compbench_eval_numeracy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e1466e9857ababe"}},{"code_sha256_prefix":"4d1bebaf5c2d7d4c","entry":"spline_interpolation","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"dot/compbench_eval_motion_binding.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/dot/compbench_eval_motion_binding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4d1bebaf5c2d7d4c"}},{"code_sha256_prefix":"d893c774b1f34088","entry":"generate_caption","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"Grounded-Segment-Anything/automatic_label_demo.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/Grounded-Segment-Anything/automatic_label_demo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d893c774b1f34088"}},{"code_sha256_prefix":"c0648fc5a77afd71","entry":"generate_tags","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"Grounded-Segment-Anything/automatic_label_demo.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/Grounded-Segment-Anything/automatic_label_demo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c0648fc5a77afd71"}},{"code_sha256_prefix":"4552e857b0cf7ef7","entry":"load_image","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"LLaVA/llava/eval/compbench_eval_action_binding.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/LLaVA/llava/eval/compbench_eval_action_binding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4552e857b0cf7ef7"}},{"code_sha256_prefix":"99d66c86efac69ff","entry":"load_images","repo":"KaiyueSun98/T2V-CompBench","repo_kind":"official","path":"LLaVA/llava/eval/compbench_eval_action_binding.py","file_url":"https://github.com/KaiyueSun98/T2V-CompBench/blob/HEAD/LLaVA/llava/eval/compbench_eval_action_binding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99d66c86efac69ff"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}