{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diffusion-2-dynamic-3d-content-generation-via","title":"Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models","arxiv_id":"2404.02148","date":"2024-04-02","proceeding":null,"authors":["Zeyu Yang","Zijie Pan","Chun Gu","Li Zhang"],"abstract":"Recent advancements in 3D generation are predominantly propelled by improvements in 3D-aware image diffusion models. These models are pretrained on Internet-scale image data and fine-tuned on massive 3D data, offering the capability of producing highly consistent multi-view images. However, due to the scarcity of synchronized multi-view video data, it remains challenging to adapt this paradigm to 4D generation directly. Despite that, the available video and 3D data are adequate for training video and multi-view diffusion models separately that can provide satisfactory dynamic and geometric priors respectively. To take advantage of both, this paper presents Diffusion$^2$, a novel framework for dynamic 3D content creation that reconciles the knowledge about geometric consistency and temporal smoothness from these models to directly sample dense multi-view multi-frame images which can be employed to optimize continuous 4D representation. Specifically, we design a simple yet effective denoising strategy via score composition of pretrained video and multi-view diffusion models based on the probability structure of the target image array. To alleviate the potential conflicts between two heterogeneous scores, we further introduce variance-reducing sampling via interpolated steps, facilitating smooth and stable generation. Owing to the high parallelism of the proposed image generation process and the efficiency of the modern 4D reconstruction pipeline, our framework can generate 4D content within few minutes. Notably, our method circumvents the reliance on expensive and hard-to-scale 4D data, thereby having the potential to benefit from the scaling of the foundation video and multi-view diffusion models. Extensive experiments demonstrate the efficacy of our proposed framework in generating highly seamless and consistent 4D assets under various types of conditions.","url_abs":"https://arxiv.org/abs/2404.02148v4","url_pdf":"https://arxiv.org/pdf/2404.02148v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diffusion-2-dynamic-3d-content-generation-via","repo_url":"https://github.com/fudan-zvg/diffusion-square","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"3d-generation","task_name":"3D Generation"},{"task_slug":"4d-reconstruction","task_name":"4D reconstruction"},{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.02148","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.02148"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fudan-zvg/diffusion-square","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"ran_violates":3,"ran":7,"ran_fixture":1},"by_repo_kind":{"official":{"samples":11,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"9fcdaa6e423e8aa7","entry":"Normalize","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"9fcdaa6e423e8aa7"}},{"code_sha256_prefix":"424012cb37b31172","entry":"default","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/modules/attention.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"424012cb37b31172"}},{"code_sha256_prefix":"4cb732f513d69dfd","entry":"disabled_train","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/util.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/util.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cb732f513d69dfd"}},{"code_sha256_prefix":"fefdc1dcee5c4986","entry":"do_sample","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/inference/helpers.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/inference/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fefdc1dcee5c4986"}},{"code_sha256_prefix":"aa5486a3650902d8","entry":"exists","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/modules/attention.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5486a3650902d8"}},{"code_sha256_prefix":"c52b1e32cdda91f9","entry":"get_batch","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/inference/helpers.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/inference/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c52b1e32cdda91f9"}},{"code_sha256_prefix":"327b8b8283ecfd73","entry":"get_discretization_config","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/inference/api.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/inference/api.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"327b8b8283ecfd73"}},{"code_sha256_prefix":"f89b6fdf331885c2","entry":"get_guider_config","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/inference/api.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/inference/api.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f89b6fdf331885c2"}},{"code_sha256_prefix":"2036522d462af205","entry":"get_string_from_tuple","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/util.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2036522d462af205"}},{"code_sha256_prefix":"cb49209c125de1b4","entry":"get_timestep_embedding","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"cb49209c125de1b4"}},{"code_sha256_prefix":"6686c8acf02ec831","entry":"get_unique_embedder_keys_from_conditioner","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/inference/helpers.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/inference/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6686c8acf02ec831"}},{"code_sha256_prefix":"3753e92206c00bb9","entry":"is_power_of_two","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/util.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3753e92206c00bb9"}},{"code_sha256_prefix":"3137073275f8c21a","entry":"nonlinearity","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"3137073275f8c21a"}},{"code_sha256_prefix":"9a299fe5ae09e407","entry":"uniq","repo":"fudan-zvg/diffusion-square","repo_kind":"official","path":"sgm/modules/attention.py","file_url":"https://github.com/fudan-zvg/diffusion-square/blob/HEAD/sgm/modules/attention.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a299fe5ae09e407"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}