{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/comparing-bad-apples-to-good-oranges-aligning","title":"Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization","arxiv_id":"2404.00530","date":"2024-03-31","proceeding":null,"authors":["Hritik Bansal","Ashima Suvarna","Gantavya Bhatt","Nanyun Peng","Kai-Wei Chang","Aditya Grover"],"abstract":"A common technique for aligning large language models (LLMs) relies on acquiring human preferences by comparing multiple generations conditioned on a fixed context. This method, however, relies solely on pairwise comparisons, where the generations are evaluated within an identical context. While effective to such conditional preferences often fail to encompass the nuanced and multidimensional nature of human preferences. In this work, we revisit the traditional paradigm of preference acquisition and propose a new axis based on eliciting preferences jointly over the instruction-response pairs. Unlike prior preference optimizations, which are designed for conditional ranking protocols (e.g., DPO), we propose Joint Preference Optimization (JPO), a new preference optimization objective that upweights the joint probability of the chosen instruction-response pair over the rejected instruction-response pair. Interestingly, LLMs trained with joint instruction-response preference data using JPO outperform LLM trained with DPO by $5.2\\%$ and $3.3\\%$ win-rate for summarization and open-ended dialogue datasets, respectively. Our findings reveal that joint preferences over instruction and response pairs can significantly enhance the alignment of LLMs by tapping into a broader spectrum of human preference elicitation. The data and code is available at https://github.com/Hritikbansal/dove.","url_abs":"https://arxiv.org/abs/2404.00530v2","url_pdf":"https://arxiv.org/pdf/2404.00530v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"comparing-bad-apples-to-good-oranges-aligning","repo_url":"https://github.com/hritikbansal/dove","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"dpo","method_name":"DPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.00530","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.00530"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hritikbansal/dove","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":11},"by_repo_kind":{"official":{"samples":11,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cee7785fcfddc8ae","entry":"allocate","repo":"hritikbansal/dove","repo_kind":"official","path":"sft/core/multipack_sampler.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/sft/core/multipack_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cee7785fcfddc8ae"}},{"code_sha256_prefix":"5c2e6ebfd6b20c74","entry":"ffd_check","repo":"hritikbansal/dove","repo_kind":"official","path":"sft/core/multipack_sampler.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/sft/core/multipack_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c2e6ebfd6b20c74"}},{"code_sha256_prefix":"d0405e95f05e1f64","entry":"ffd_with_result","repo":"hritikbansal/dove","repo_kind":"official","path":"sft/core/multipack_sampler.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/sft/core/multipack_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0405e95f05e1f64"}},{"code_sha256_prefix":"f0832ee83b5ff758","entry":"filter_long_samples","repo":"hritikbansal/dove","repo_kind":"official","path":"sft/core/supervised_dataset.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/sft/core/supervised_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f0832ee83b5ff758"}},{"code_sha256_prefix":"df4f346585f694bb","entry":"filter_long_sequences","repo":"hritikbansal/dove","repo_kind":"official","path":"jpo/utils.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/jpo/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df4f346585f694bb"}},{"code_sha256_prefix":"d12d009f6f16a459","entry":"fmt_prompt","repo":"hritikbansal/dove","repo_kind":"official","path":"jpo/utils.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/jpo/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d12d009f6f16a459"}},{"code_sha256_prefix":"dc2a8a5f4eb6f801","entry":"fmt_prompt","repo":"hritikbansal/dove","repo_kind":"official","path":"sft/core/supervised_dataset.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/sft/core/supervised_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dc2a8a5f4eb6f801"}},{"code_sha256_prefix":"841ccc2439e64906","entry":"get_feedback","repo":"hritikbansal/dove","repo_kind":"official","path":"jpo/ai_feedback.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/jpo/ai_feedback.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"841ccc2439e64906"}},{"code_sha256_prefix":"4abdf461254ec7ab","entry":"get_feedback","repo":"hritikbansal/dove","repo_kind":"official","path":"jpo/auto_eval.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/jpo/auto_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4abdf461254ec7ab"}},{"code_sha256_prefix":"f1d264774d681d67","entry":"preprocess","repo":"hritikbansal/dove","repo_kind":"official","path":"sft/core/supervised_dataset.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/sft/core/supervised_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1d264774d681d67"}},{"code_sha256_prefix":"129ed5f0939a11d7","entry":"return_prompt_and_responses_augmented","repo":"hritikbansal/dove","repo_kind":"official","path":"jpo/utils.py","file_url":"https://github.com/hritikbansal/dove/blob/HEAD/jpo/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"129ed5f0939a11d7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}