{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dreamphysics-learning-physical-properties-of","title":"DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors","arxiv_id":"2406.01476","date":"2024-06-03","proceeding":null,"authors":["Tianyu Huang","Haoze Zhang","Yihan Zeng","Zhilu Zhang","Hui Li","WangMeng Zuo","Rynson W. H. Lau"],"abstract":"Dynamic 3D interaction has been attracting a lot of attention recently. However, creating such 4D content remains challenging. One solution is to animate 3D scenes with physics-based simulation, which requires manually assigning precise physical properties to the object or the simulated results would become unnatural. Another solution is to learn the deformation of 3D objects with the distillation of video generative models, which, however, tends to produce 3D videos with small and discontinuous motions due to the inappropriate extraction and application of physics priors. In this work, to combine the strengths and complementing shortcomings of the above two solutions, we propose to learn the physical properties of a material field with video diffusion priors, and then utilize a physics-based Material-Point-Method (MPM) simulator to generate 4D content with realistic motions. In particular, we propose motion distillation sampling to emphasize video motion information during distillation. In addition, to facilitate the optimization, we further propose a KAN-based material field with frame boosting. Experimental results demonstrate that our method enjoys more realistic motions than state-of-the-arts do.","url_abs":"https://arxiv.org/abs/2406.01476v3","url_pdf":"https://arxiv.org/pdf/2406.01476v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dreamphysics-learning-physical-properties-of","repo_url":"https://github.com/tyhuang0428/dreamphysics","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.01476","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.01476"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tyhuang0428/dreamphysics","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"fbdcb6c5792c5d8c","entry":"decode_param_json","repo":"tyhuang0428/dreamphysics","repo_kind":"official","path":"utils/decode_param.py","file_url":"https://github.com/tyhuang0428/dreamphysics/blob/HEAD/utils/decode_param.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbdcb6c5792c5d8c"}},{"code_sha256_prefix":"ddd567b7a94424d3","entry":"generate_local_coord","repo":"tyhuang0428/dreamphysics","repo_kind":"official","path":"utils/camera_view_utils.py","file_url":"https://github.com/tyhuang0428/dreamphysics/blob/HEAD/utils/camera_view_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddd567b7a94424d3"}},{"code_sha256_prefix":"2b9cf8baab6bc8b4","entry":"generate_camera_rotation_matrix","repo":"tyhuang0428/dreamphysics","repo_kind":"official","path":"utils/camera_view_utils.py","file_url":"https://github.com/tyhuang0428/dreamphysics/blob/HEAD/utils/camera_view_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b9cf8baab6bc8b4"}},{"code_sha256_prefix":"fc9200f0d0d5f797","entry":"get_point_on_sphere","repo":"tyhuang0428/dreamphysics","repo_kind":"official","path":"utils/camera_view_utils.py","file_url":"https://github.com/tyhuang0428/dreamphysics/blob/HEAD/utils/camera_view_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc9200f0d0d5f797"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}