{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-world-simulator-crafting-physical","title":"Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation","arxiv_id":"2410.05363","date":"2024-10-07","proceeding":null,"authors":["Fanqing Meng","Jiaqi Liao","Xinyu Tan","Wenqi Shao","Quanfeng Lu","Kaipeng Zhang","Yu Cheng","Dianqi Li","Yu Qiao","Ping Luo"],"abstract":"Text-to-video (T2V) models like Sora have made significant strides in visualizing complex prompts, which is increasingly viewed as a promising path towards constructing the universal world simulator. Cognitive psychologists believe that the foundation for achieving this goal is the ability to understand intuitive physics. However, the capacity of these models to accurately represent intuitive physics remains largely unexplored. To bridge this gap, we introduce PhyGenBench, a comprehensive \\textbf{Phy}sics \\textbf{Gen}eration \\textbf{Ben}chmark designed to evaluate physical commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully crafted prompts across 27 distinct physical laws, spanning four fundamental domains, which could comprehensively assesses models' understanding of physical commonsense. Alongside PhyGenBench, we propose a novel evaluation framework called PhyGenEval. This framework employs a hierarchical evaluation structure utilizing appropriate advanced vision-language models and large language models to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can conduct large-scale automated assessments of T2V models' understanding of physical commonsense, which align closely with human feedback. Our evaluation results and in-depth analysis demonstrate that current models struggle to generate videos that comply with physical commonsense. Moreover, simply scaling up models or employing prompt engineering techniques is insufficient to fully address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We hope this study will inspire the community to prioritize the learning of physical commonsense in these models beyond entertainment applications. We will release the data and codes at https://github.com/OpenGVLab/PhyGenBench","url_abs":"https://arxiv.org/abs/2410.05363v1","url_pdf":"https://arxiv.org/pdf/2410.05363v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-world-simulator-crafting-physical","repo_url":"https://github.com/opengvlab/phygenbench","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"},{"task_slug":"video-generation","task_name":"Video Generation"}],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.05363","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05363"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/opengvlab/phygenbench","reach":{"status":"ok"}}],"summary":{"ran_violates":1,"ran_draft_wrong":1,"ran":9,"unverified":2},"by_repo_kind":{"official":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"8204634e93f03f47","entry":"calculate_average_score","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/video/MTScore/InternVideo_physical.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/video/MTScore/InternVideo_physical.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8204634e93f03f47"}},{"code_sha256_prefix":"f41cb1a19b154297","entry":"encode_image","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/multi/GPT4o.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/multi/GPT4o.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f41cb1a19b154297"}},{"code_sha256_prefix":"6493211c704e0b4c","entry":"encode_images","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/multi/GPT4o.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/multi/GPT4o.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6493211c704e0b4c"}},{"code_sha256_prefix":"c4af2e63b61c4766","entry":"eval_string","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/video/MTScore/configs/config.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/video/MTScore/configs/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c4af2e63b61c4766"}},{"code_sha256_prefix":"f7510953d3aad7af","entry":"extract_json","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/semantic/gpt4o_sementic.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/semantic/gpt4o_sementic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7510953d3aad7af"}},{"code_sha256_prefix":"ae81ceb0833d3443","entry":"get_video_frame_count","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/single/vqascore.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/single/vqascore.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae81ceb0833d3443"}},{"code_sha256_prefix":"6955bc871b9bbeb3","entry":"maybe_align","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/video/MTScore/layer_norm.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/video/MTScore/layer_norm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6955bc871b9bbeb3"}},{"code_sha256_prefix":"e41cae3b88c4eedd","entry":"merge_a_into_b","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/video/MTScore/configs/config.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/video/MTScore/configs/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e41cae3b88c4eedd"}},{"code_sha256_prefix":"9e9a9aca86fc6043","entry":"process_video","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/video/GPT4o.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/video/GPT4o.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e9a9aca86fc6043"}},{"code_sha256_prefix":"f7dcf1d5ccde53d1","entry":"sample_frames","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/single/vqascore.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/single/vqascore.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7dcf1d5ccde53d1"}},{"code_sha256_prefix":"0e4947868f47c8f0","entry":"setup_openai_client","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/semantic/gpt4o_sementic.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/semantic/gpt4o_sementic.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e4947868f47c8f0"}},{"code_sha256_prefix":"348649b748fc17a8","entry":"call_gpt4","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/semantic/gpt4o_sementic.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/semantic/gpt4o_sementic.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"348649b748fc17a8"}},{"code_sha256_prefix":"91bc1b8fd39b1a58","entry":"eval_dict_leaf","repo":"opengvlab/phygenbench","repo_kind":"official","path":"PhyGenEval/video/MTScore/configs/config.py","file_url":"https://github.com/opengvlab/phygenbench/blob/HEAD/PhyGenEval/video/MTScore/configs/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91bc1b8fd39b1a58"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}