{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dancegrpo-unleashing-grpo-on-visual","title":"DanceGRPO: Unleashing GRPO on Visual Generation","arxiv_id":"2505.07818","date":"2025-05-12","proceeding":null,"authors":["Zeyue Xue","Jie Wu","Yu Gao","Fangyuan Kong","Lingting Zhu","Mengzhao Chen","Zhiheng Liu","Wei Liu","Qiushan Guo","Weilin Huang","Ping Luo"],"abstract":"Recent breakthroughs in generative models-particularly diffusion models and rectified flows-have revolutionized visual content creation, yet aligning model outputs with human preferences remains a critical challenge. Existing reinforcement learning (RL)-based methods for visual generation face critical limitations: incompatibility with modern Ordinary Differential Equations (ODEs)-based sampling paradigms, instability in large-scale training, and lack of validation for video generation. This paper introduces DanceGRPO, the first unified framework to adapt Group Relative Policy Optimization (GRPO) to visual generation paradigms, unleashing one unified RL algorithm across two generative paradigms (diffusion models and rectified flows), three tasks (text-to-image, text-to-video, image-to-video), four foundation models (Stable Diffusion, HunyuanVideo, FLUX, SkyReel-I2V), and five reward models (image/video aesthetics, text-image alignment, video motion quality, and binary reward). To our knowledge, DanceGRPO is the first RL-based unified framework capable of seamless adaptation across diverse generative paradigms, tasks, foundational models, and reward models. DanceGRPO demonstrates consistent and substantial improvements, which outperform baselines by up to 181% on benchmarks such as HPS-v2.1, CLIP Score, VideoAlign, and GenEval. Notably, DanceGRPO not only can stabilize policy optimization for complex video generation, but also enables generative policy to better capture denoising trajectories for Best-of-N inference scaling and learn from sparse binary feedback. Our results establish DanceGRPO as a robust and versatile solution for scaling Reinforcement Learning from Human Feedback (RLHF) tasks in visual generation, offering new insights into harmonizing reinforcement learning and visual synthesis. The code will be released.","url_abs":"https://arxiv.org/abs/2505.07818v1","url_pdf":"https://arxiv.org/pdf/2505.07818v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dancegrpo-unleashing-grpo-on-visual","repo_url":"https://github.com/xuezeyue/dancegrpo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"denoising","task_name":"Denoising"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"video-generation","task_name":"Video Generation"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.07818","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.07818"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xuezeyue/dancegrpo","reach":null}],"summary":{"ran_fixture":1,"ran_honours":1,"ran_violates":1,"unverified":1},"by_repo_kind":{"listed":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"fe8a641669476f3e","entry":"flux_step","repo":"xuezeyue/dancegrpo","repo_kind":"listed","path":"fastvideo/train_grpo_flux.py","file_url":"https://github.com/xuezeyue/dancegrpo/blob/HEAD/fastvideo/train_grpo_flux.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe8a641669476f3e"}},{"code_sha256_prefix":"4080cb55d923dfba","entry":"prepare_latent_image_ids","repo":"xuezeyue/dancegrpo","repo_kind":"listed","path":"fastvideo/train_grpo_flux.py","file_url":"https://github.com/xuezeyue/dancegrpo/blob/HEAD/fastvideo/train_grpo_flux.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4080cb55d923dfba"}},{"code_sha256_prefix":"a34c4529d41a7731","entry":"sd3_time_shift","repo":"xuezeyue/dancegrpo","repo_kind":"listed","path":"fastvideo/train_grpo_flux.py","file_url":"https://github.com/xuezeyue/dancegrpo/blob/HEAD/fastvideo/train_grpo_flux.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a34c4529d41a7731"}},{"code_sha256_prefix":"cdafd4faaa0baf71","entry":"run_sample_step","repo":"xuezeyue/dancegrpo","repo_kind":"listed","path":"fastvideo/train_grpo_hunyuan.py","file_url":"https://github.com/xuezeyue/dancegrpo/blob/HEAD/fastvideo/train_grpo_hunyuan.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdafd4faaa0baf71"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}