{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adding-conditional-control-to-diffusion","title":"Adding Conditional Control to Diffusion Models with Reinforcement Learning","arxiv_id":"2406.12120","date":"2024-06-17","proceeding":null,"authors":["Yulai Zhao","Masatoshi Uehara","Gabriele Scalia","Tommaso Biancalani","Sergey Levine","Ehsan Hajiramezanali"],"abstract":"Diffusion models are powerful generative models that allow for precise control over the characteristics of the generated samples. While these diffusion models trained on large datasets have achieved success, there is often a need to introduce additional controls in downstream fine-tuning processes, treating these powerful models as pre-trained diffusion models. This work presents a novel method based on reinforcement learning (RL) to add additional controls, leveraging an offline dataset comprising inputs and corresponding labels. We formulate this task as an RL problem, with the classifier learned from the offline dataset and the KL divergence against pre-trained models serving as the reward functions. We introduce our method, $\\textbf{CTRL}$ ($\\textbf{C}$onditioning pre-$\\textbf{T}$rained diffusion models with $\\textbf{R}$einforcement $\\textbf{L}$earning), which produces soft-optimal policies that maximize the abovementioned reward functions. We formally demonstrate that our method enables sampling from the conditional distribution conditioned on additional controls during inference. Our RL-based approach offers several advantages over existing methods. Compared to commonly used classifier-free guidance, our approach improves sample efficiency, and can greatly simplify offline dataset construction by exploiting conditional independence between the inputs and additional controls. Furthermore, unlike classifier guidance, we avoid the need to train classifiers from intermediate states to additional controls.","url_abs":"https://arxiv.org/abs/2406.12120v1","url_pdf":"https://arxiv.org/pdf/2406.12120v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.12120","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.12120"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/zhaoyl18/CTRL","reach":null}],"summary":{"ran":4,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":9,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"72d310c425a7148c","entry":"MLPDiff_class","repo":"zhaoyl18/CTRL","repo_kind":"found_in_text","path":"condition_main.py","file_url":"https://github.com/zhaoyl18/CTRL/blob/HEAD/condition_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"72d310c425a7148c"}},{"code_sha256_prefix":"8b9ee4524a67745d","entry":"ResidualBlock","repo":"zhaoyl18/CTRL","repo_kind":"found_in_text","path":"condition_main.py","file_url":"https://github.com/zhaoyl18/CTRL/blob/HEAD/condition_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8b9ee4524a67745d"}},{"code_sha256_prefix":"45c9c8b6addda684","entry":"TemperatureScaler","repo":"zhaoyl18/CTRL","repo_kind":"found_in_text","path":"condition_main.py","file_url":"https://github.com/zhaoyl18/CTRL/blob/HEAD/condition_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"45c9c8b6addda684"}},{"code_sha256_prefix":"3379e9ac5c94ea0b","entry":"ThreeLayerConvNet","repo":"zhaoyl18/CTRL","repo_kind":"found_in_text","path":"condition_main.py","file_url":"https://github.com/zhaoyl18/CTRL/blob/HEAD/condition_main.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3379e9ac5c94ea0b"}},{"code_sha256_prefix":"b3e82c8411307ba6","entry":"condition_AestheticScorerDiff","repo":"zhaoyl18/CTRL","repo_kind":"found_in_text","path":"condition_main.py","file_url":"https://github.com/zhaoyl18/CTRL/blob/HEAD/condition_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b3e82c8411307ba6"}},{"code_sha256_prefix":"1992dadaf00954ff","entry":"condition_CompressibilityScorerDiff","repo":"zhaoyl18/CTRL","repo_kind":"found_in_text","path":"condition_main.py","file_url":"https://github.com/zhaoyl18/CTRL/blob/HEAD/condition_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1992dadaf00954ff"}},{"code_sha256_prefix":"a949161676497e63","entry":"condition_aesthetic_loss_fn","repo":"zhaoyl18/CTRL","repo_kind":"found_in_text","path":"condition_main.py","file_url":"https://github.com/zhaoyl18/CTRL/blob/HEAD/condition_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a949161676497e63"}},{"code_sha256_prefix":"a7e8abe24d6b3f60","entry":"condition_compressibility_loss_fn","repo":"zhaoyl18/CTRL","repo_kind":"found_in_text","path":"condition_main.py","file_url":"https://github.com/zhaoyl18/CTRL/blob/HEAD/condition_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a7e8abe24d6b3f60"}},{"code_sha256_prefix":"87f15babe5022bcc","entry":"multitask_loss_fn","repo":"zhaoyl18/CTRL","repo_kind":"found_in_text","path":"condition_main.py","file_url":"https://github.com/zhaoyl18/CTRL/blob/HEAD/condition_main.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"87f15babe5022bcc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}