{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unicontrol-a-unified-diffusion-model-for-1","title":"UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild","arxiv_id":"2305.11147","date":"2023-05-18","proceeding":"NeurIPS 2023 11","authors":["Can Qin","Shu Zhang","Ning Yu","Yihao Feng","Xinyi Yang","Yingbo Zhou","Huan Wang","Juan Carlos Niebles","Caiming Xiong","Silvio Savarese","Stefano Ermon","Yun Fu","ran Xu"],"abstract":"Achieving machine autonomy and human control often represent divergent objectives in the design of interactive AI systems. Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages. However, they often fall short in generating images with spatial, structural, or geometric controls. The integration of such controls, which can accommodate various visual conditions in a single unified model, remains an unaddressed challenge. In response, we introduce UniControl, a new generative foundation model that consolidates a wide array of controllable condition-to-image (C2I) tasks within a singular framework, while still allowing for arbitrary language prompts. UniControl enables pixel-level-precise image generation, where visual conditions primarily influence the generated structures and language prompts guide the style and context. To equip UniControl with the capacity to handle diverse visual conditions, we augment pretrained text-to-image diffusion models and introduce a task-aware HyperNet to modulate the diffusion models, enabling the adaptation to different C2I tasks simultaneously. Trained on nine unique C2I tasks, UniControl demonstrates impressive zero-shot generation abilities with unseen visual conditions. Experimental results show that UniControl often surpasses the performance of single-task-controlled methods of comparable model sizes. This control versatility positions UniControl as a significant advancement in the realm of controllable visual generation.","url_abs":"https://arxiv.org/abs/2305.11147v3","url_pdf":"https://arxiv.org/pdf/2305.11147v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unicontrol-a-unified-diffusion-model-for-1","repo_url":"https://github.com/salesforce/unicontrol","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.11147","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.11147"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/salesforce/unicontrol","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran_violates":1,"unverified":5},"by_repo_kind":{"official":{"samples":7,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0dbf855dab1ab575","entry":"get_node_name","repo":"salesforce/unicontrol","repo_kind":"official","path":"tool_add_control.py","file_url":"https://github.com/salesforce/unicontrol/blob/HEAD/tool_add_control.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0dbf855dab1ab575"}},{"code_sha256_prefix":"1e63d588563eb90a","entry":"numpy_to_pil","repo":"salesforce/unicontrol","repo_kind":"official","path":"utils.py","file_url":"https://github.com/salesforce/unicontrol/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e63d588563eb90a"}},{"code_sha256_prefix":"cfc17707f35e7ec4","entry":"get_state_dict","repo":"salesforce/unicontrol","repo_kind":"official","path":"cldm/model.py","file_url":"https://github.com/salesforce/unicontrol/blob/HEAD/cldm/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cfc17707f35e7ec4"}},{"code_sha256_prefix":"d044cb7f97b33225","entry":"load_replacement","repo":"salesforce/unicontrol","repo_kind":"official","path":"utils.py","file_url":"https://github.com/salesforce/unicontrol/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d044cb7f97b33225"}},{"code_sha256_prefix":"df733a879693145d","entry":"load_state_dict","repo":"salesforce/unicontrol","repo_kind":"official","path":"cldm/model.py","file_url":"https://github.com/salesforce/unicontrol/blob/HEAD/cldm/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df733a879693145d"}},{"code_sha256_prefix":"6f66e41f8b1a278d","entry":"modulated_conv2d","repo":"salesforce/unicontrol","repo_kind":"official","path":"cldm/cldm_unicontrol.py","file_url":"https://github.com/salesforce/unicontrol/blob/HEAD/cldm/cldm_unicontrol.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f66e41f8b1a278d"}},{"code_sha256_prefix":"81718137a49d87b1","entry":"put_watermark","repo":"salesforce/unicontrol","repo_kind":"official","path":"utils.py","file_url":"https://github.com/salesforce/unicontrol/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81718137a49d87b1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}