{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/direct-preference-optimization-with-an-offset","title":"Direct Preference Optimization with an Offset","arxiv_id":"2402.10571","date":"2024-02-16","proceeding":null,"authors":["Afra Amini","Tim Vieira","Ryan Cotterell"],"abstract":"Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. For instance, if a response contains harmful or toxic content, the annotator will have a strong preference for that response. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.","url_abs":"https://arxiv.org/abs/2402.10571v2","url_pdf":"https://arxiv.org/pdf/2402.10571v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"direct-preference-optimization-with-an-offset","repo_url":"https://github.com/rycolab/odpo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"direct-preference-optimization-with-an-offset","repo_url":"https://github.com/xxares/todo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"dpo","method_name":"DPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.10571","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.10571"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xxares/todo","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rycolab/odpo","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":6,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"12f6803880f9b17e","entry":"extract_anthropic_prompt","repo":"rycolab/odpo","repo_kind":"official","path":"preference_datasets.py","file_url":"https://github.com/rycolab/odpo/blob/HEAD/preference_datasets.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"12f6803880f9b17e"}},{"code_sha256_prefix":"10ab71f27ffd7de8","entry":"get_local_run_dir","repo":"rycolab/odpo","repo_kind":"official","path":"utils.py","file_url":"https://github.com/rycolab/odpo/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10ab71f27ffd7de8"}},{"code_sha256_prefix":"86c5aaf696fc2128","entry":"preference_loss","repo":"rycolab/odpo","repo_kind":"official","path":"trainers.py","file_url":"https://github.com/rycolab/odpo/blob/HEAD/trainers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86c5aaf696fc2128"}},{"code_sha256_prefix":"8ab43db002bf984d","entry":"get_local_dir","repo":"rycolab/odpo","repo_kind":"official","path":"utils.py","file_url":"https://github.com/rycolab/odpo/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ab43db002bf984d"}},{"code_sha256_prefix":"2e4ab4d5b3f6d465","entry":"get_pos_sentiment_rewards","repo":"rycolab/odpo","repo_kind":"official","path":"imdb_sentiment.py","file_url":"https://github.com/rycolab/odpo/blob/HEAD/imdb_sentiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e4ab4d5b3f6d465"}},{"code_sha256_prefix":"e7ce4c19888c6df6","entry":"get_remote_file","repo":"rycolab/odpo","repo_kind":"official","path":"utils.py","file_url":"https://github.com/rycolab/odpo/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7ce4c19888c6df6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}