{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/direct-preference-based-policy-optimization-1","title":"Direct Preference-based Policy Optimization without Reward Modeling","arxiv_id":"2301.12842","date":"2023-01-30","proceeding":"NeurIPS 2023 11","authors":["Gaon An","Junhyeok Lee","Xingdong Zuo","Norio Kosaka","Kyung-Min Kim","Hyun Oh Song"],"abstract":"Preference-based reinforcement learning (PbRL) is an approach that enables RL agents to learn from preference, which is particularly useful when formulating a reward function is challenging. Existing PbRL methods generally involve a two-step procedure: they first learn a reward model based on given preference data and then employ off-the-shelf reinforcement learning algorithms using the learned reward model. However, obtaining an accurate reward model solely from preference information, especially when the preference is from human teachers, can be difficult. Instead, we propose a PbRL algorithm that directly learns from preference without requiring any reward modeling. To achieve this, we adopt a contrastive learning framework to design a novel policy scoring metric that assigns a high score to policies that align with the given preferences. We apply our algorithm to offline RL tasks with actual human preference labels and show that our algorithm outperforms or is on par with the existing PbRL methods. Notably, on high-dimensional control tasks, our algorithm surpasses offline RL methods that learn with ground-truth reward information. Finally, we show that our algorithm can be successfully applied to fine-tune large language models.","url_abs":"https://arxiv.org/abs/2301.12842v3","url_pdf":"https://arxiv.org/pdf/2301.12842v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"direct-preference-based-policy-optimization-1","repo_url":"https://github.com/snu-mllab/dppo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"direct-preference-based-policy-optimization-1","repo_url":"https://github.com/jhejna/cpl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"offline-rl","task_name":"Offline RL"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"contrastive-learning","method_name":"Contrastive Learning"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2301.12842","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2301.12842"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/snu-mllab/DPPO","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/snu-mllab/dppo","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jhejna/cpl","reach":null}],"summary":{"ran_honours":1,"ran":3,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"20922c79a0929bbf","entry":"flatten","repo":"snu-mllab/DPPO","repo_kind":"official","path":"viskit/core.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/viskit/core.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20922c79a0929bbf"}},{"code_sha256_prefix":"718462716d78ea99","entry":"default_init","repo":"snu-mllab/DPPO","repo_kind":"official","path":"common.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/common.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"718462716d78ea99"}},{"code_sha256_prefix":"965e75a1245dd833","entry":"load_progress","repo":"snu-mllab/DPPO","repo_kind":"official","path":"viskit/core.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/viskit/core.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"965e75a1245dd833"}},{"code_sha256_prefix":"18c566cc009ddb07","entry":"multiple_action_q_function","repo":"snu-mllab/DPPO","repo_kind":"official","path":"JaxPref/model.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/JaxPref/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18c566cc009ddb07"}},{"code_sha256_prefix":"6c7a4a9349a395e9","entry":"unique","repo":"snu-mllab/DPPO","repo_kind":"official","path":"viskit/core.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/viskit/core.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6c7a4a9349a395e9"}},{"code_sha256_prefix":"79158e16a5bf1c29","entry":"batch_to_jax","repo":"snu-mllab/DPPO","repo_kind":"official","path":"JaxPref/dataset_utils.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/JaxPref/dataset_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79158e16a5bf1c29"}},{"code_sha256_prefix":"566d9b710e08c1ee","entry":"evaluate","repo":"snu-mllab/DPPO","repo_kind":"official","path":"evaluation.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"566d9b710e08c1ee"}},{"code_sha256_prefix":"7fbb5020f7fab8ff","entry":"qlearning_kitchen_dataset","repo":"snu-mllab/DPPO","repo_kind":"official","path":"JaxPref/human_label_preprocess.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/JaxPref/human_label_preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fbb5020f7fab8ff"}},{"code_sha256_prefix":"ec250e16f078c140","entry":"qlearning_mujoco_adroit_dataset","repo":"snu-mllab/DPPO","repo_kind":"official","path":"JaxPref/human_label_preprocess.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/JaxPref/human_label_preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec250e16f078c140"}},{"code_sha256_prefix":"908dca4182b46323","entry":"safe_norm","repo":"snu-mllab/DPPO","repo_kind":"official","path":"learner.py","file_url":"https://github.com/snu-mllab/DPPO/blob/HEAD/learner.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"908dca4182b46323"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}