{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-better-alignment-training-diffusion","title":"Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards","arxiv_id":"2503.11240","date":"2025-03-14","proceeding":"CVPR 2025 1","authors":["Zijing Hu","Fengda Zhang","Long Chen","Kun Kuang","Jiahui Li","Kaifeng Gao","Jun Xiao","Xin Wang","Wenwu Zhu"],"abstract":"Diffusion models have achieved remarkable success in text-to-image generation. However, their practical applications are hindered by the misalignment between generated images and corresponding text prompts. To tackle this issue, reinforcement learning (RL) has been considered for diffusion model fine-tuning. Yet, RL's effectiveness is limited by the challenge of sparse reward, where feedback is only available at the end of the generation process. This makes it difficult to identify which actions during the denoising process contribute positively to the final generated image, potentially leading to ineffective or unnecessary denoising policies. To this end, this paper presents a novel RL-based framework that addresses the sparse reward problem when training diffusion models. Our framework, named $\\text{B}^2\\text{-DiffuRL}$, employs two strategies: \\textbf{B}ackward progressive training and \\textbf{B}ranch-based sampling. For one thing, backward progressive training focuses initially on the final timesteps of denoising process and gradually extends the training interval to earlier timesteps, easing the learning difficulty from sparse rewards. For another, we perform branch-based sampling for each training interval. By comparing the samples within the same branch, we can identify how much the policies of the current training interval contribute to the final image, which helps to learn effective policies instead of unnecessary ones. $\\text{B}^2\\text{-DiffuRL}$ is compatible with existing optimization algorithms. Extensive experiments demonstrate the effectiveness of $\\text{B}^2\\text{-DiffuRL}$ in improving prompt-image alignment and maintaining diversity in generated images. The code for this work is available.","url_abs":"https://arxiv.org/abs/2503.11240v2","url_pdf":"https://arxiv.org/pdf/2503.11240v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-better-alignment-training-diffusion","repo_url":"https://github.com/hu-zijing/b2-diffurl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"image-generation","task_name":"Image Generation"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"text-to-image-generation-1","task_name":"Text to Image Generation"},{"task_slug":"text-to-image-generation","task_name":"Text-to-Image Generation"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.11240","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.11240"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hu-zijing/b2-diffurl","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/hu-zijing/B2-DiffuRL","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2595903a62fcd29c","entry":"ddim_step_with_logprob","repo":"hu-zijing/B2-DiffuRL","repo_kind":"official","path":"diffusion/ddim_with_logprob.py","file_url":"https://github.com/hu-zijing/B2-DiffuRL/blob/HEAD/diffusion/ddim_with_logprob.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2595903a62fcd29c"}},{"code_sha256_prefix":"d270df0ea806963a","entry":"latents_decode","repo":"hu-zijing/B2-DiffuRL","repo_kind":"official","path":"diffusion/ddim_with_logprob.py","file_url":"https://github.com/hu-zijing/B2-DiffuRL/blob/HEAD/diffusion/ddim_with_logprob.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d270df0ea806963a"}},{"code_sha256_prefix":"4d6d69f9827e1e27","entry":"load_data_from_json","repo":"hu-zijing/B2-DiffuRL","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/hu-zijing/B2-DiffuRL/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d6d69f9827e1e27"}},{"code_sha256_prefix":"17bb604273b271d0","entry":"load_sample","repo":"hu-zijing/B2-DiffuRL","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/hu-zijing/B2-DiffuRL/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"17bb604273b271d0"}},{"code_sha256_prefix":"ae70b34aca0bb7d0","entry":"load_sample_stage","repo":"hu-zijing/B2-DiffuRL","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/hu-zijing/B2-DiffuRL/blob/HEAD/utils/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae70b34aca0bb7d0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}