{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-efficient-and-exact-optimization-of","title":"Towards Efficient Exact Optimization of Language Model Alignment","arxiv_id":"2402.00856","date":"2024-02-01","proceeding":null,"authors":["Haozhe Ji","Cheng Lu","Yilin Niu","Pei Ke","Hongning Wang","Jun Zhu","Jie Tang","Minlie Huang"],"abstract":"The alignment of language models with human preferences is vital for their application in real-world tasks. The problem is formulated as optimizing the model's policy to maximize the expected reward that reflects human preferences with minimal deviation from the initial policy. While considered as a straightforward solution, reinforcement learning (RL) suffers from high variance in policy updates, which impedes efficient policy improvement. Recently, direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. However, we show that DPO derived based on the optimal solution of the problem leads to a compromised mean-seeking approximation of the optimal solution in practice. In this paper, we propose efficient exact optimization (EXO) of the alignment objective. EXO is guaranteed to optimize in the same direction as RL algorithms asymptotically for arbitrary policy parametrization. This leads to the same mode-seeking solution, while enables efficient optimization by circumventing the complexities of RL. We also compare our method to DPO with both theoretical and empirical analyses, and further demonstrate the advantages of our method over existing approaches on realistic human preference data. Code is available at https://github.com/haozheji/exact-optimization.","url_abs":"https://arxiv.org/abs/2402.00856v4","url_pdf":"https://arxiv.org/pdf/2402.00856v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-efficient-and-exact-optimization-of","repo_url":"https://github.com/haozheji/exact-optimization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"towards-efficient-and-exact-optimization-of","repo_url":"https://github.com/neverix/rlhf-trojan-2024-cod","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"dpo","method_name":"DPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.00856","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.00856"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/neverix/rlhf-trojan-2024-cod","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/haozheji/exact-optimization","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a01fc02f147c8243","entry":"load_data","repo":"haozheji/exact-optimization","repo_kind":"official","path":"src/align_stage/calc_logprobs.py","file_url":"https://github.com/haozheji/exact-optimization/blob/HEAD/src/align_stage/calc_logprobs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a01fc02f147c8243"}},{"code_sha256_prefix":"ba3d77f8bc434527","entry":"load_json","repo":"haozheji/exact-optimization","repo_kind":"official","path":"src/align_stage/eval_kl.py","file_url":"https://github.com/haozheji/exact-optimization/blob/HEAD/src/align_stage/eval_kl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba3d77f8bc434527"}},{"code_sha256_prefix":"f87ec21e6cc554fb","entry":"dpo_loss","repo":"haozheji/exact-optimization","repo_kind":"official","path":"src/align_stage/loss.py","file_url":"https://github.com/haozheji/exact-optimization/blob/HEAD/src/align_stage/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f87ec21e6cc554fb"}},{"code_sha256_prefix":"ca139c7b291fafb7","entry":"exact_loss","repo":"haozheji/exact-optimization","repo_kind":"official","path":"src/align_stage/loss.py","file_url":"https://github.com/haozheji/exact-optimization/blob/HEAD/src/align_stage/loss.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca139c7b291fafb7"}},{"code_sha256_prefix":"af991fecf57a358d","entry":"get_data_path","repo":"haozheji/exact-optimization","repo_kind":"official","path":"src/align_stage/inference.py","file_url":"https://github.com/haozheji/exact-optimization/blob/HEAD/src/align_stage/inference.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"af991fecf57a358d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}