{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unpacking-dpo-and-ppo-disentangling-best","title":"Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback","arxiv_id":"2406.09279","date":"2024-06-13","proceeding":null,"authors":["Hamish Ivison","Yizhong Wang","Jiacheng Liu","Zeqiu Wu","Valentina Pyatkin","Nathan Lambert","Noah A. Smith","Yejin Choi","Hannaneh Hajishirzi"],"abstract":"Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models (LMs). Despite its widespread use, the way preference-based learning is applied varies wildly, with differing data, learning algorithms, and evaluations used, making disentangling the impact of each aspect difficult. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts, systematically investigate the impact of these components on downstream model performance, and suggest a recipe for strong learning for preference feedback. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements, followed by the choice of learning algorithm, the use of improved reward models, and finally the use of additional unlabeled prompts for policy training. Notably, PPO outperforms DPO by up to 2.5% in math and 1.2% in general domains. High-quality preference data leads to improvements of up to 8% in instruction following and truthfulness. Despite significant gains of up to 5% in mathematical evaluation when scaling up reward models, we surprisingly observe marginal improvements in other categories. We publicly release the code used for training (https://github.com/hamishivi/EasyLM) and evaluating (https://github.com/allenai/open-instruct) our models, along with the models and datasets themselves (https://huggingface.co/collections/allenai/tulu-v25-suite-66676520fd578080e126f618).","url_abs":"https://arxiv.org/abs/2406.09279v2","url_pdf":"https://arxiv.org/pdf/2406.09279v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unpacking-dpo-and-ppo-disentangling-best","repo_url":"https://github.com/allenai/open-instruct","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"unpacking-dpo-and-ppo-disentangling-best","repo_url":"https://github.com/hamishivi/easylm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"dpo","method_name":"DPO"},{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.09279","atlas_url":"https://app.syntology.ai/?focus=2406.09279","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.09279"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/hamishivi/EasyLM","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/allenai/open-instruct","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hamishivi/easylm","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"ran_fixture":3,"ran_draft_wrong":4,"unverified":4},"by_repo_kind":{"official":{"samples":19,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f14840c40641281","entry":"apply_rotary_pos_emb","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/models/gptj/gptj_model.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/models/gptj/gptj_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0f14840c40641281"}},{"code_sha256_prefix":"dbfe03f34fca42fc","entry":"blockwise_attn","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/bpt.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/bpt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dbfe03f34fca42fc"}},{"code_sha256_prefix":"119ac1062aa47fe1","entry":"concatenated_forward","repo":"hamishivi/EasyLM","repo_kind":"official","path":"EasyLM/models/llama/llama_train_dpo.py","file_url":"https://github.com/hamishivi/EasyLM/blob/HEAD/EasyLM/models/llama/llama_train_dpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"119ac1062aa47fe1"}},{"code_sha256_prefix":"4c4036a5cbd39f10","entry":"convert_logits_to_logps","repo":"hamishivi/EasyLM","repo_kind":"official","path":"EasyLM/models/llama/llama_train_dpo.py","file_url":"https://github.com/hamishivi/EasyLM/blob/HEAD/EasyLM/models/llama/llama_train_dpo.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c4036a5cbd39f10"}},{"code_sha256_prefix":"ff574bc6676f34f1","entry":"create_sinusoidal_positions","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/models/gptj/gptj_model.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/models/gptj/gptj_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff574bc6676f34f1"}},{"code_sha256_prefix":"0cad657acf954c2f","entry":"get_jax_mesh","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/jax_utils.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/jax_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0cad657acf954c2f"}},{"code_sha256_prefix":"dad72196bdb2301a","entry":"get_names_from_parition_spec","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/jax_utils.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/jax_utils.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dad72196bdb2301a"}},{"code_sha256_prefix":"7125896060f86225","entry":"masked_mean","repo":"hamishivi/EasyLM","repo_kind":"official","path":"EasyLM/models/llama/llama_train_ppo.py","file_url":"https://github.com/hamishivi/EasyLM/blob/HEAD/EasyLM/models/llama/llama_train_ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7125896060f86225"}},{"code_sha256_prefix":"7cbbe7f38c74cab8","entry":"masked_sum","repo":"hamishivi/EasyLM","repo_kind":"official","path":"EasyLM/models/llama/llama_train_ppo.py","file_url":"https://github.com/hamishivi/EasyLM/blob/HEAD/EasyLM/models/llama/llama_train_ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7cbbe7f38c74cab8"}},{"code_sha256_prefix":"c39dac404ff3f2b3","entry":"masked_var","repo":"hamishivi/EasyLM","repo_kind":"official","path":"EasyLM/models/llama/llama_train_ppo.py","file_url":"https://github.com/hamishivi/EasyLM/blob/HEAD/EasyLM/models/llama/llama_train_ppo.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c39dac404ff3f2b3"}},{"code_sha256_prefix":"cc2ba1b1b0c83f6c","entry":"match_keywords","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/models/llama/convert_easylm_to_hf.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/models/llama/convert_easylm_to_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc2ba1b1b0c83f6c"}},{"code_sha256_prefix":"1db35ae97b8e784f","entry":"optax_add_scheduled_weight_decay","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/optimizers.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/optimizers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1db35ae97b8e784f"}},{"code_sha256_prefix":"cdf684770f174015","entry":"pad_out_to_full_batch","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/data.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cdf684770f174015"}},{"code_sha256_prefix":"c5bcf01d18bba63d","entry":"read_json","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/models/llama/convert_easylm_to_hf.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/models/llama/convert_easylm_to_hf.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c5bcf01d18bba63d"}},{"code_sha256_prefix":"768565e8f56c5fb0","entry":"rotate_every_two","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/models/gptj/gptj_model.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/models/gptj/gptj_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"768565e8f56c5fb0"}},{"code_sha256_prefix":"4833ab3506125762","entry":"blockwise_ffn","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/bpt.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/bpt.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4833ab3506125762"}},{"code_sha256_prefix":"a6d706f6e0699b5c","entry":"dpo_loss","repo":"hamishivi/EasyLM","repo_kind":"official","path":"EasyLM/models/llama/llama_train_dpo.py","file_url":"https://github.com/hamishivi/EasyLM/blob/HEAD/EasyLM/models/llama/llama_train_dpo.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a6d706f6e0699b5c"}},{"code_sha256_prefix":"667a8648b1f3ad2c","entry":"inverse_permute","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/models/llama/convert_hf_to_easylm.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/models/llama/convert_hf_to_easylm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"667a8648b1f3ad2c"}},{"code_sha256_prefix":"e23d17afdb6ab8b8","entry":"make_shard_and_gather_fns","repo":"hamishivi/easylm","repo_kind":"official","path":"EasyLM/jax_utils.py","file_url":"https://github.com/hamishivi/easylm/blob/HEAD/EasyLM/jax_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e23d17afdb6ab8b8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}