{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/beyond-reverse-kl-generalizing-direct","title":"Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints","arxiv_id":"2309.16240","date":"2023-09-28","proceeding":null,"authors":["Chaoqi Wang","Yibo Jiang","Chenghao Yang","Han Liu","Yuxin Chen"],"abstract":"The increasing capabilities of large language models (LLMs) raise opportunities for artificial general intelligence but concurrently amplify safety concerns, such as potential misuse of AI systems, necessitating effective AI alignment. Reinforcement Learning from Human Feedback (RLHF) has emerged as a promising pathway towards AI alignment but brings forth challenges due to its complexity and dependence on a separate reward model. Direct Preference Optimization (DPO) has been proposed as an alternative, and it remains equivalent to RLHF under the reverse KL regularization constraint. This paper presents $f$-DPO, a generalized approach to DPO by incorporating diverse divergence constraints. We show that under certain $f$-divergences, including Jensen-Shannon divergence, forward KL divergences and $\\alpha$-divergences, the complex relationship between the reward and optimal policy can also be simplified by addressing the Karush-Kuhn-Tucker conditions. This eliminates the need for estimating the normalizing constant in the Bradley-Terry model and enables a tractable mapping between the reward function and the optimal policy. Our approach optimizes LLMs to align with human preferences in a more efficient and supervised manner under a broad set of divergence constraints. Empirically, adopting these divergences ensures a balance between alignment performance and generation diversity. Importantly, $f$-DPO outperforms PPO-based methods in divergence efficiency, and divergence constraints directly influence expected calibration error (ECE).","url_abs":"https://arxiv.org/abs/2309.16240v1","url_pdf":"https://arxiv.org/pdf/2309.16240v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"beyond-reverse-kl-generalizing-direct","repo_url":"https://github.com/alecwangcq/f-divergence-dpo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"dpo","method_name":"DPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.16240","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.16240"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alecwangcq/f-divergence-dpo","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":4,"unverified":12},"by_repo_kind":{"listed":{"samples":17,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"12f6803880f9b17e","entry":"extract_anthropic_prompt","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"preference_datasets.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/preference_datasets.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12f6803880f9b17e"}},{"code_sha256_prefix":"10ab71f27ffd7de8","entry":"get_local_run_dir","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10ab71f27ffd7de8"}},{"code_sha256_prefix":"41fd9954784a1330","entry":"load_judge_prompts","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"mt_bench/common.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/mt_bench/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41fd9954784a1330"}},{"code_sha256_prefix":"270d3eb4371b0d85","entry":"load_model_answers","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"mt_bench/common.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/mt_bench/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"270d3eb4371b0d85"}},{"code_sha256_prefix":"88969cdc675c418e","entry":"load_questions","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"mt_bench/common.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/mt_bench/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"88969cdc675c418e"}},{"code_sha256_prefix":"194f8dc522933267","entry":"calculate_entropy","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"generation.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"194f8dc522933267"}},{"code_sha256_prefix":"f2074220eaa2a95d","entry":"calculate_score","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"cali/hh_cali_eval.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/cali/hh_cali_eval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f2074220eaa2a95d"}},{"code_sha256_prefix":"c38b6fc85679b0b0","entry":"calculate_self_bleu","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"generation.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/generation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c38b6fc85679b0b0"}},{"code_sha256_prefix":"fc5f3e7e6f97a222","entry":"dpo_loss","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"trainers.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/trainers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc5f3e7e6f97a222"}},{"code_sha256_prefix":"6ccfbad96ffcc246","entry":"forward_dpo_loss","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"trainers.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/trainers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ccfbad96ffcc246"}},{"code_sha256_prefix":"d24ec6691ff04146","entry":"get_judge_name","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"mt_bench/compute_agreement.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/mt_bench/compute_agreement.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d24ec6691ff04146"}},{"code_sha256_prefix":"8ab43db002bf984d","entry":"get_local_dir","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ab43db002bf984d"}},{"code_sha256_prefix":"7745c0e675406cf2","entry":"get_mt_bench_votes_data","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"mt_bench/compute_agreement.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/mt_bench/compute_agreement.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7745c0e675406cf2"}},{"code_sha256_prefix":"e7ce4c19888c6df6","entry":"get_remote_file","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7ce4c19888c6df6"}},{"code_sha256_prefix":"3e9f2dd8699a6fff","entry":"load_model","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"mt_bench/convert_dpo_trainer_file_to_huggingface.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/mt_bench/convert_dpo_trainer_file_to_huggingface.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e9f2dd8699a6fff"}},{"code_sha256_prefix":"35801bf32affe123","entry":"revert","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"mt_bench/compute_agreement.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/mt_bench/compute_agreement.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"35801bf32affe123"}},{"code_sha256_prefix":"82acf8c8f8bc3e2b","entry":"stabilized_log1pexp","repo":"alecwangcq/f-divergence-dpo","repo_kind":"listed","path":"trainers.py","file_url":"https://github.com/alecwangcq/f-divergence-dpo/blob/HEAD/trainers.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"82acf8c8f8bc3e2b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}