{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/controlling-text-to-image-diffusion-by","title":"Controlling Text-to-Image Diffusion by Orthogonal Finetuning","arxiv_id":"2306.07280","date":"2023-06-12","proceeding":"NeurIPS 2023 11","authors":["Zeju Qiu","Weiyang Liu","Haiwen Feng","Yuxuan Xue","Yao Feng","Zhen Liu","Dan Zhang","Adrian Weller","Bernhard Schölkopf"],"abstract":"Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We find that this property is crucial for preserving the semantic generation ability of text-to-image diffusion models. To improve finetuning stability, we further propose Constrained Orthogonal Finetuning (COFT) which imposes an additional radius constraint to the hypersphere. Specifically, we consider two important finetuning text-to-image tasks: subject-driven generation where the goal is to generate subject-specific images given a few images of a subject and a text prompt, and controllable generation where the goal is to enable the model to take in additional control signals. We empirically show that our OFT framework outperforms existing methods in generation quality and convergence speed.","url_abs":"https://arxiv.org/abs/2306.07280v3","url_pdf":"https://arxiv.org/pdf/2306.07280v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"controlling-text-to-image-diffusion-by","repo_url":"https://github.com/zeju1997/oft","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"controlling-text-to-image-diffusion-by","repo_url":"https://github.com/zqiu24/oft","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2306.07280","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.07280"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zeju1997/oft","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zqiu24/oft","reach":null}],"summary":{"ran":1,"ran_draft_wrong":4,"unverified":17},"by_repo_kind":{"listed":{"samples":22,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e85418d3d1fc5999","entry":"OFTInjectedLinear","repo":"zqiu24/oft","repo_kind":"listed","path":"oft-control/oft.py","file_url":"https://github.com/zqiu24/oft/blob/HEAD/oft-control/oft.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e85418d3d1fc5999"}},{"code_sha256_prefix":"229d634611bdc645","entry":"collate_fn","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/hyperparameter_oft.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/hyperparameter_oft.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"229d634611bdc645"}},{"code_sha256_prefix":"0dbf855dab1ab575","entry":"get_node_name","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-control/tool_add_control_oft.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-control/tool_add_control_oft.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0dbf855dab1ab575"}},{"code_sha256_prefix":"ccec8c1b07c1a3b6","entry":"project","repo":"zqiu24/oft","repo_kind":"listed","path":"oft-control/oft.py","file_url":"https://github.com/zqiu24/oft/blob/HEAD/oft-control/oft.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ccec8c1b07c1a3b6"}},{"code_sha256_prefix":"4cd1d6571b9e35a1","entry":"project_batch","repo":"zqiu24/oft","repo_kind":"listed","path":"oft-control/oft.py","file_url":"https://github.com/zqiu24/oft/blob/HEAD/oft-control/oft.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4cd1d6571b9e35a1"}},{"code_sha256_prefix":"a73565d61cbda080","entry":"calculate_metrics","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-control/eval_canny.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-control/eval_canny.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a73565d61cbda080"}},{"code_sha256_prefix":"862b8dc1d24afa80","entry":"clip_image","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/eval_ablation.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/eval_ablation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"862b8dc1d24afa80"}},{"code_sha256_prefix":"028c10fb01328ebd","entry":"clip_text","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/eval_ablation.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/eval_ablation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"028c10fb01328ebd"}},{"code_sha256_prefix":"8c1f9091bcfcc35f","entry":"dino","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/eval_ablation.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/eval_ablation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c1f9091bcfcc35f"}},{"code_sha256_prefix":"bca0e14a517d825c","entry":"find_directories_missing_folders","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/check_empty.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/check_empty.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bca0e14a517d825c"}},{"code_sha256_prefix":"33aebbf50ad78324","entry":"find_subdirectory_names_without_png","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/check_empty.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/check_empty.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33aebbf50ad78324"}},{"code_sha256_prefix":"6431708d8f3ff177","entry":"get_full_repo_name","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/train_dreambooth_flax.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/train_dreambooth_flax.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6431708d8f3ff177"}},{"code_sha256_prefix":"320666d58c95245c","entry":"get_params_to_save","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/train_dreambooth_flax.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/train_dreambooth_flax.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"320666d58c95245c"}},{"code_sha256_prefix":"cfc17707f35e7ec4","entry":"get_state_dict","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-control/oldm/model.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-control/oldm/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfc17707f35e7ec4"}},{"code_sha256_prefix":"2f00b39db5a81466","entry":"import_model_class_from_model_name_or_path","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/hyperparameter_oft.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/hyperparameter_oft.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f00b39db5a81466"}},{"code_sha256_prefix":"5cfa761d3c77fa2d","entry":"inject_trainable_oft","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-control/oft.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-control/oft.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cfa761d3c77fa2d"}},{"code_sha256_prefix":"df733a879693145d","entry":"load_state_dict","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-control/oldm/model.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-control/oldm/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df733a879693145d"}},{"code_sha256_prefix":"9a1b8f02d12708b3","entry":"parse_args","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/hyperparameter_oft.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/hyperparameter_oft.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a1b8f02d12708b3"}},{"code_sha256_prefix":"f5132e29bee34d4d","entry":"parse_args","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/train_dreambooth.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/train_dreambooth.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5132e29bee34d4d"}},{"code_sha256_prefix":"2f3af25e8d9d78f5","entry":"parse_args","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-db/train_dreambooth_lora.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-db/train_dreambooth_lora.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2f3af25e8d9d78f5"}},{"code_sha256_prefix":"1ff5174d4258a8ab","entry":"plot_kpts","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-control/eval_landmark.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-control/eval_landmark.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1ff5174d4258a8ab"}},{"code_sha256_prefix":"9320471cb08ee42e","entry":"plot_points","repo":"zeju1997/oft","repo_kind":"listed","path":"oft-control/eval_landmark.py","file_url":"https://github.com/zeju1997/oft/blob/HEAD/oft-control/eval_landmark.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9320471cb08ee42e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}