{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/evalcrafter-benchmarking-and-evaluating-large","title":"EvalCrafter: Benchmarking and Evaluating Large Video Generation Models","arxiv_id":"2310.11440","date":"2023-10-17","proceeding":"CVPR 2024 1","authors":["Yaofang Liu","Xiaodong Cun","Xuebo Liu","Xintao Wang","Yong Zhang","Haoxin Chen","Yang Liu","Tieyong Zeng","Raymond Chan","Ying Shan"],"abstract":"The vision and language generative models have been overgrown in recent years. For video generation, various open-sourced models and public-available services have been developed to generate high-quality videos. However, these methods often use a few metrics, e.g., FVD or IS, to evaluate the performance. We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Thus, we propose a novel framework and pipeline for exhaustively evaluating the performance of the generated videos. Our approach involves generating a diverse and comprehensive list of 700 prompts for text-to-video generation, which is based on an analysis of real-world user data and generated with the assistance of a large language model. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmark, in terms of visual qualities, content qualities, motion qualities, and text-video alignment with 17 well-selected objective metrics. To obtain the final leaderboard of the models, we further fit a series of coefficients to align the objective metrics to the users' opinions. Based on the proposed human alignment method, our final score shows a higher correlation than simply averaging the metrics, showing the effectiveness of the proposed evaluation method.","url_abs":"https://arxiv.org/abs/2310.11440v3","url_pdf":"https://arxiv.org/pdf/2310.11440v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"evalcrafter-benchmarking-and-evaluating-large","repo_url":"https://github.com/EvalCrafter/EvalCrafter","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"text-to-video-generation","task_name":"Text-to-Video Generation"},{"task_slug":"video-alignment","task_name":"Video Alignment"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[{"slug":"evalcrafter-text-to-video-ectv-dataset","name":"EvalCrafter Text-to-Video (ECTV) Dataset","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.11440","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.11440"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/EvalCrafter/EvalCrafter","reach":{"status":"ok"}}],"summary":{"ran":5,"unverified":6},"by_repo_kind":{"official":{"samples":11,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"5ef767b6b97ea4a9","entry":"compute_max","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/RAFT/Scores_with_CLIP_release.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/RAFT/Scores_with_CLIP_release.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ef767b6b97ea4a9"}},{"code_sha256_prefix":"0e67fae78e6dd37c","entry":"format_time","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/dnnlib/util.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/dnnlib/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e67fae78e6dd37c"}},{"code_sha256_prefix":"fbe06db04a182567","entry":"read_prompt_from_txt","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/is.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/is.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbe06db04a182567"}},{"code_sha256_prefix":"afc709e407f0b256","entry":"read_text_file","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/ocr_score.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/ocr_score.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"afc709e407f0b256"}},{"code_sha256_prefix":"900a8d68bf48579e","entry":"tuple_product","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/dnnlib/util.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/dnnlib/util.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"900a8d68bf48579e"}},{"code_sha256_prefix":"9d31d2c4cd16bb2d","entry":"ask_yes_no","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/dnnlib/util.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/dnnlib/util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d31d2c4cd16bb2d"}},{"code_sha256_prefix":"c04d29af305ec9b6","entry":"calculate_clip_score","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/RAFT/Scores_with_CLIP_release.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/RAFT/Scores_with_CLIP_release.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c04d29af305ec9b6"}},{"code_sha256_prefix":"7bff88199cee559a","entry":"calculate_clip_temp_score","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/RAFT/Scores_with_CLIP_release.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/RAFT/Scores_with_CLIP_release.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7bff88199cee559a"}},{"code_sha256_prefix":"6e4c34d736239a45","entry":"get_inception_feature","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/pytorch_gan_metrics/core.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/pytorch_gan_metrics/core.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e4c34d736239a45"}},{"code_sha256_prefix":"2dfa8886e774088c","entry":"sqrt_newton_schulz","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/pytorch_gan_metrics/core.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/pytorch_gan_metrics/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2dfa8886e774088c"}},{"code_sha256_prefix":"7f018aa22ac761c4","entry":"torch_cov","repo":"EvalCrafter/EvalCrafter","repo_kind":"official","path":"metrics/pytorch_gan_metrics/core.py","file_url":"https://github.com/EvalCrafter/EvalCrafter/blob/HEAD/metrics/pytorch_gan_metrics/core.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7f018aa22ac761c4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}