{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/controllable-text-to-3d-generation-via","title":"Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting","arxiv_id":"2403.09981","date":"2024-03-15","proceeding":null,"authors":["Zhiqi Li","Yiming Chen","Lingzhe Zhao","Peidong Liu"],"abstract":"While text-to-3D and image-to-3D generation tasks have received considerable attention, one important but under-explored field between them is controllable text-to-3D generation, which we mainly focus on in this work. To address this task, 1) we introduce Multi-view ControlNet (MVControl), a novel neural network architecture designed to enhance existing pre-trained multi-view diffusion models by integrating additional input conditions, such as edge, depth, normal, and scribble maps. Our innovation lies in the introduction of a conditioning module that controls the base diffusion model using both local and global embeddings, which are computed from the input condition images and camera poses. Once trained, MVControl is able to offer 3D diffusion guidance for optimization-based 3D generation. And, 2) we propose an efficient multi-stage 3D generation pipeline that leverages the benefits of recent large reconstruction models and score distillation algorithm. Building upon our MVControl architecture, we employ a unique hybrid diffusion guidance method to direct the optimization process. In pursuit of efficiency, we adopt 3D Gaussians as our representation instead of the commonly used implicit representations. We also pioneer the use of SuGaR, a hybrid representation that binds Gaussians to mesh triangle faces. This approach alleviates the issue of poor geometry in 3D Gaussians and enables the direct sculpting of fine-grained geometry on the mesh. Extensive experiments demonstrate that our method achieves robust generalization and enables the controllable generation of high-quality 3D content.","url_abs":"https://arxiv.org/abs/2403.09981v2","url_pdf":"https://arxiv.org/pdf/2403.09981v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"controllable-text-to-3d-generation-via","repo_url":"https://github.com/WU-CVGL/MVControl-threestudio","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"controllable-text-to-3d-generation-via","repo_url":"https://github.com/wu-cvgl/mvcontrol","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"3d-generation","task_name":"3D Generation"},{"task_slug":"image-to-3d","task_name":"Image to 3D"},{"task_slug":"text-to-3d","task_name":"Text to 3D"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2403.09981","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.09981"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/wu-cvgl/mvcontrol","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/WU-CVGL/MVControl-threestudio","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":1,"ran_fixture":1,"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1},"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"05595953e2bf10ee","entry":"load_custom_module","repo":"wu-cvgl/mvcontrol","repo_kind":"listed","path":"launch.py","file_url":"https://github.com/wu-cvgl/mvcontrol/blob/HEAD/launch.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"05595953e2bf10ee"}},{"code_sha256_prefix":"e20f2847f4726ebc","entry":"pad_to_square","repo":"wu-cvgl/mvcontrol","repo_kind":"listed","path":"app_stage1.py","file_url":"https://github.com/wu-cvgl/mvcontrol/blob/HEAD/app_stage1.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e20f2847f4726ebc"}},{"code_sha256_prefix":"8d6e850be8ffb9fc","entry":"pred_x0","repo":"WU-CVGL/MVControl-threestudio","repo_kind":"official","path":"threestudio/models/guidance/mvcontrol_guidance.py","file_url":"https://github.com/WU-CVGL/MVControl-threestudio/blob/HEAD/threestudio/models/guidance/mvcontrol_guidance.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8d6e850be8ffb9fc"}},{"code_sha256_prefix":"74039a42b74173e3","entry":"tv_loss","repo":"WU-CVGL/MVControl-threestudio","repo_kind":"official","path":"threestudio/utils/loss.py","file_url":"https://github.com/WU-CVGL/MVControl-threestudio/blob/HEAD/threestudio/utils/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74039a42b74173e3"}},{"code_sha256_prefix":"da94debb8019ad46","entry":"zero_module","repo":"WU-CVGL/MVControl-threestudio","repo_kind":"official","path":"extern/mvcontrol/mvcontrolnet.py","file_url":"https://github.com/WU-CVGL/MVControl-threestudio/blob/HEAD/extern/mvcontrol/mvcontrolnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da94debb8019ad46"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}