{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/aligning-diffusion-behaviors-with-q-functions","title":"Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control","arxiv_id":"2407.09024","date":"2024-07-12","proceeding":null,"authors":["Huayu Chen","Kaiwen Zheng","Hang Su","Jun Zhu"],"abstract":"Drawing upon recent advances in language model alignment, we formulate offline Reinforcement Learning as a two-stage optimization problem: First pretraining expressive generative policies on reward-free behavior datasets, then fine-tuning these policies to align with task-specific annotations like Q-values. This strategy allows us to leverage abundant and diverse behavior data to enhance generalization and enable rapid adaptation to downstream tasks using minimal annotations. In particular, we introduce Efficient Diffusion Alignment (EDA) for solving continuous control problems. EDA utilizes diffusion models for behavior modeling. However, unlike previous approaches, we represent diffusion policies as the derivative of a scalar neural network with respect to action inputs. This representation is critical because it enables direct density calculation for diffusion models, making them compatible with existing LLM alignment theories. During policy fine-tuning, we extend preference-based alignment methods like Direct Preference Optimization (DPO) to align diffusion behaviors with continuous Q-functions. Our evaluation on the D4RL benchmark shows that EDA exceeds all baseline methods in overall performance. Notably, EDA maintains about 95\\% of performance and still outperforms several baselines given only 1\\% of Q-labelled data during fine-tuning.","url_abs":"https://arxiv.org/abs/2407.09024v2","url_pdf":"https://arxiv.org/pdf/2407.09024v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"aligning-diffusion-behaviors-with-q-functions","repo_url":"https://github.com/thu-ml/efficient-diffusion-alignment","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"continuous-control","task_name":"Continuous Control"},{"task_slug":"d4rl","task_name":"D4RL"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"continuous-control","task_name":"continuous-control"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.09024","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.09024"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/thu-ml/Efficient-Diffusion-Alignment","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/thu-ml/efficient-diffusion-alignment","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":4,"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":12,"samples":[{"code_sha256_prefix":"6a35b62fbc80f70a","entry":"interpolate_fn","repo":"thu-ml/Efficient-Diffusion-Alignment","repo_kind":"official","path":"dpm_solver_pytorch.py","file_url":"https://github.com/thu-ml/Efficient-Diffusion-Alignment/blob/HEAD/dpm_solver_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6a35b62fbc80f70a"}},{"code_sha256_prefix":"579d77f78c03773e","entry":"asymmetric_l2_loss","repo":"thu-ml/Efficient-Diffusion-Alignment","repo_kind":"official","path":"BDiffusion.py","file_url":"https://github.com/thu-ml/Efficient-Diffusion-Alignment/blob/HEAD/BDiffusion.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"579d77f78c03773e"}},{"code_sha256_prefix":"e6110366588c5c65","entry":"expand_dims","repo":"thu-ml/Efficient-Diffusion-Alignment","repo_kind":"official","path":"dpm_solver_pytorch.py","file_url":"https://github.com/thu-ml/Efficient-Diffusion-Alignment/blob/HEAD/dpm_solver_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e6110366588c5c65"}},{"code_sha256_prefix":"c8b1ea2dbf064b81","entry":"inf_train_gen","repo":"thu-ml/Efficient-Diffusion-Alignment","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/thu-ml/Efficient-Diffusion-Alignment/blob/HEAD/dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c8b1ea2dbf064b81"}},{"code_sha256_prefix":"057c095634d04249","entry":"marginal_prob_std","repo":"thu-ml/Efficient-Diffusion-Alignment","repo_kind":"official","path":"utils.py","file_url":"https://github.com/thu-ml/Efficient-Diffusion-Alignment/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"057c095634d04249"}},{"code_sha256_prefix":"d466e7f1c4364a0b","entry":"mlp","repo":"thu-ml/Efficient-Diffusion-Alignment","repo_kind":"official","path":"model.py","file_url":"https://github.com/thu-ml/Efficient-Diffusion-Alignment/blob/HEAD/model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d466e7f1c4364a0b"}},{"code_sha256_prefix":"c38c4cd486f7a766","entry":"model_wrapper","repo":"thu-ml/Efficient-Diffusion-Alignment","repo_kind":"official","path":"dpm_solver_pytorch.py","file_url":"https://github.com/thu-ml/Efficient-Diffusion-Alignment/blob/HEAD/dpm_solver_pytorch.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c38c4cd486f7a766"}},{"code_sha256_prefix":"732209d369d824c0","entry":"return_range","repo":"thu-ml/Efficient-Diffusion-Alignment","repo_kind":"official","path":"dataset.py","file_url":"https://github.com/thu-ml/Efficient-Diffusion-Alignment/blob/HEAD/dataset.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"732209d369d824c0"}},{"code_sha256_prefix":"3a044b5fa091a33b","entry":"CEP_Critic","repo":"thu-ml/efficient-diffusion-alignment","repo_kind":"official","path":"BDiffusion.py","file_url":"https://github.com/thu-ml/efficient-diffusion-alignment/blob/HEAD/BDiffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3a044b5fa091a33b"}},{"code_sha256_prefix":"8f226f7f0a3a49f9","entry":"EDA_policy","repo":"thu-ml/efficient-diffusion-alignment","repo_kind":"official","path":"BDiffusion.py","file_url":"https://github.com/thu-ml/efficient-diffusion-alignment/blob/HEAD/BDiffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f226f7f0a3a49f9"}},{"code_sha256_prefix":"b9ad37c17cb01b28","entry":"IQL_Critic","repo":"thu-ml/efficient-diffusion-alignment","repo_kind":"official","path":"BDiffusion.py","file_url":"https://github.com/thu-ml/efficient-diffusion-alignment/blob/HEAD/BDiffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b9ad37c17cb01b28"}},{"code_sha256_prefix":"5e582540e0a940ea","entry":"update_target","repo":"thu-ml/efficient-diffusion-alignment","repo_kind":"official","path":"BDiffusion.py","file_url":"https://github.com/thu-ml/efficient-diffusion-alignment/blob/HEAD/BDiffusion.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e582540e0a940ea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}