{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/triple-preference-optimization-achieving","title":"Triple Preference Optimization: Achieving Better Alignment with Less Data in a Single Step Optimization","arxiv_id":"2405.16681","date":"2024-05-26","proceeding":null,"authors":["Amir Saeidi","Shivanshu Verma","Aswin RRV","Chitta Baral"],"abstract":"Large Language Models (LLMs) perform well across diverse tasks, but aligning them with human demonstrations is challenging. Recently, Reinforcement Learning (RL)-free methods like Direct Preference Optimization (DPO) have emerged, offering improved stability and scalability while retaining competitive performance relative to RL-based methods. However, while RL-free methods deliver satisfactory performance, they require significant data to develop a robust Supervised Fine-Tuned (SFT) model and an additional step to fine-tune this model on a preference dataset, which constrains their utility and scalability. In this paper, we introduce Triple Preference Optimization (TPO), a new preference learning method designed to align an LLM with three preferences without requiring a separate SFT step and using considerably less data. Through a combination of practical experiments and theoretical analysis, we show the efficacy of TPO as a single-step alignment strategy. Specifically, we fine-tuned the Phi-2 (2.7B) and Mistral (7B) models using TPO directly on the UltraFeedback dataset, achieving superior results compared to models aligned through other methods such as SFT, DPO, KTO, IPO, CPO, and ORPO. Moreover, the performance of TPO without the SFT component led to notable improvements in the MT-Bench score, with increases of +1.27 and +0.63 over SFT and DPO, respectively. Additionally, TPO showed higher average accuracy, surpassing DPO and SFT by 4.2% and 4.97% on the Open LLM Leaderboard benchmarks. Our code is publicly available at https://github.com/sahsaeedi/triple-preference-optimization .","url_abs":"https://arxiv.org/abs/2405.16681v1","url_pdf":"https://arxiv.org/pdf/2405.16681v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"triple-preference-optimization-achieving","repo_url":"https://github.com/sahsaeedi/triple-preference-optimization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"dpo","method_name":"DPO"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.16681","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.16681"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sahsaeedi/triple-preference-optimization","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":4,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a9cb92f38a00fa54","entry":"apply_chat_template","repo":"sahsaeedi/triple-preference-optimization","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/sahsaeedi/triple-preference-optimization/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9cb92f38a00fa54"}},{"code_sha256_prefix":"dfc8e2556a70f4f3","entry":"compute_accuracy","repo":"sahsaeedi/triple-preference-optimization","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/sahsaeedi/triple-preference-optimization/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dfc8e2556a70f4f3"}},{"code_sha256_prefix":"d75c549406e58a87","entry":"get_global_statistics","repo":"sahsaeedi/triple-preference-optimization","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/sahsaeedi/triple-preference-optimization/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d75c549406e58a87"}},{"code_sha256_prefix":"e78362d84c5945b0","entry":"pad","repo":"sahsaeedi/triple-preference-optimization","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/sahsaeedi/triple-preference-optimization/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e78362d84c5945b0"}},{"code_sha256_prefix":"312e65ff5e48081e","entry":"get_datasets","repo":"sahsaeedi/triple-preference-optimization","repo_kind":"official","path":"utils/data.py","file_url":"https://github.com/sahsaeedi/triple-preference-optimization/blob/HEAD/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"312e65ff5e48081e"}},{"code_sha256_prefix":"f08eeb1ca2446b4e","entry":"load_model","repo":"sahsaeedi/triple-preference-optimization","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/sahsaeedi/triple-preference-optimization/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f08eeb1ca2446b4e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}