{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ctrlora-an-extensible-and-efficient-framework","title":"CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation","arxiv_id":"2410.09400","date":"2024-10-12","proceeding":null,"authors":["Yifeng Xu","Zhenliang He","Shiguang Shan","Xilin Chen"],"abstract":"Recently, large-scale diffusion models have made impressive progress in text-to-image (T2I) generation. To further equip these T2I models with fine-grained spatial control, approaches like ControlNet introduce an extra network that learns to follow a condition image. However, for every single condition type, ControlNet requires independent training on millions of data pairs with hundreds of GPU hours, which is quite expensive and makes it challenging for ordinary users to explore and develop new types of conditions. To address this problem, we propose the CtrLoRA framework, which trains a Base ControlNet to learn the common knowledge of image-to-image generation from multiple base conditions, along with condition-specific LoRAs to capture distinct characteristics of each condition. Utilizing our pretrained Base ControlNet, users can easily adapt it to new conditions, requiring as few as 1,000 data pairs and less than one hour of single-GPU training to obtain satisfactory results in most scenarios. Moreover, our CtrLoRA reduces the learnable parameters by 90% compared to ControlNet, significantly lowering the threshold to distribute and deploy the model weights. Extensive experiments on various types of conditions demonstrate the efficiency and effectiveness of our method. Codes and model weights will be released at https://github.com/xyfJASON/ctrlora.","url_abs":"https://arxiv.org/abs/2410.09400v1","url_pdf":"https://arxiv.org/pdf/2410.09400v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ctrlora-an-extensible-and-efficient-framework","repo_url":"https://github.com/xyfjason/ctrlora","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"conditional-image-generation","task_name":"Conditional Image Generation"},{"task_slug":null,"task_name":"GPU"},{"task_slug":"image-generation","task_name":"Image Generation"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.09400","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.09400"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/xyfJASON/ctrlora","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xyfjason/ctrlora","reach":null}],"summary":{"ran":7,"ran_violates":1,"ran_draft_wrong":3,"ran_honours":2,"unverified":9},"by_repo_kind":{"official":{"samples":16,"ran":9,"repositories":1},"community":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"9da413a606c2fa21","entry":"LoRACompatibleLinear","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9da413a606c2fa21"}},{"code_sha256_prefix":"902d1f89b0cf6904","entry":"LoRALinearLayer","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"902d1f89b0cf6904"}},{"code_sha256_prefix":"f60e775a28631190","entry":"MemoryEfficientCrossAttention","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f60e775a28631190"}},{"code_sha256_prefix":"0e10d815fe77258a","entry":"QKVAttentionLegacy","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0e10d815fe77258a"}},{"code_sha256_prefix":"1fc018a678b94a86","entry":"SwitchableConv2d","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1fc018a678b94a86"}},{"code_sha256_prefix":"6be4fea6271b2108","entry":"SwitchableGroupNorm","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6be4fea6271b2108"}},{"code_sha256_prefix":"2181071c92a4cce8","entry":"SwitchableLayerNorm","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2181071c92a4cce8"}},{"code_sha256_prefix":"10ec89be27e3e44a","entry":"check_key","repo":"xyfJASON/ctrlora","repo_kind":"official","path":"app/gradio_ctrlora.py","file_url":"https://github.com/xyfJASON/ctrlora/blob/HEAD/app/gradio_ctrlora.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10ec89be27e3e44a"}},{"code_sha256_prefix":"8450074c26c4e371","entry":"get_config","repo":"xyfJASON/ctrlora","repo_kind":"official","path":"app/gradio_ctrlora.py","file_url":"https://github.com/xyfJASON/ctrlora/blob/HEAD/app/gradio_ctrlora.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8450074c26c4e371"}},{"code_sha256_prefix":"a1f6aab648f338b8","entry":"inject_trainable_lora","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"a1f6aab648f338b8"}},{"code_sha256_prefix":"4b4a8984a2737ae5","entry":"inject_trainable_lora","repo":"cloneofsimo/lora","repo_kind":"community","path":"lora_diffusion/lora.py","file_url":"https://github.com/cloneofsimo/lora/blob/HEAD/lora_diffusion/lora.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b4a8984a2737ae5"}},{"code_sha256_prefix":"5ed08f9762be44ee","entry":"inject_trainable_lora_extended","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"5ed08f9762be44ee"}},{"code_sha256_prefix":"3aec51ff694c19af","entry":"inject_trainable_lora_extended","repo":"cloneofsimo/lora","repo_kind":"community","path":"lora_diffusion/lora.py","file_url":"https://github.com/cloneofsimo/lora/blob/HEAD/lora_diffusion/lora.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3aec51ff694c19af"}},{"code_sha256_prefix":"de2f868c40cb8ebe","entry":"AttentionBlock","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"de2f868c40cb8ebe"}},{"code_sha256_prefix":"c02f4e63f9a3b275","entry":"BasicTransformerBlock","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c02f4e63f9a3b275"}},{"code_sha256_prefix":"9c1933018e1fe3c4","entry":"ControlNet","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c1933018e1fe3c4"}},{"code_sha256_prefix":"23aca6b588778459","entry":"ControlNetInference","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23aca6b588778459"}},{"code_sha256_prefix":"dd3dd6c4cfa9bb05","entry":"ResBlock","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dd3dd6c4cfa9bb05"}},{"code_sha256_prefix":"df38578e601e8130","entry":"SpatialTransformer","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df38578e601e8130"}},{"code_sha256_prefix":"b714f25e81b2ceff","entry":"TimestepEmbedSequential","repo":"xyfjason/ctrlora","repo_kind":"official","path":"cldm/cldm_ctrlora_inference.py","file_url":"https://github.com/xyfjason/ctrlora/blob/HEAD/cldm/cldm_ctrlora_inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b714f25e81b2ceff"}},{"code_sha256_prefix":"33b1ae9f923ab3ec","entry":"extract_lora_ups_down","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"33b1ae9f923ab3ec"}},{"code_sha256_prefix":"2f9880b526ff277e","entry":"extract_lora_ups_down","repo":"cloneofsimo/lora","repo_kind":"community","path":"lora_diffusion/lora.py","file_url":"https://github.com/cloneofsimo/lora/blob/HEAD/lora_diffusion/lora.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f9880b526ff277e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}