{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bridging-and-modeling-correlations-in","title":"Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization","arxiv_id":"2408.07471","date":"2024-08-14","proceeding":null,"authors":["Yuxin Jiang","Bo Huang","YuFei Wang","Xingshan Zeng","Liangyou Li","Yasheng Wang","Xin Jiang","Lifeng Shang","Ruiming Tang","Wei Wang"],"abstract":"Direct preference optimization (DPO), a widely adopted offline preference optimization algorithm, aims to align large language models (LLMs) with human-desired behaviors using pairwise preference data. However, the winning response and the losing response within pairwise data are generated isolatedly, leading to weak correlations between them as well as suboptimal alignment performance. To address this issue, we propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. Firstly, we increase the consistency and informativeness of the pairwise preference signals through targeted modifications, synthesizing a pseudo-winning response by improving the losing response with the winning response as a reference. Secondly, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations. Therefore, we propose learning token-level correlations by dynamically leveraging the policy model's confidence during training. Comprehensive experiments on QA, math, and instruction-following tasks demonstrate the effectiveness of our approach, significantly surpassing competitive baselines, including DPO. Additionally, our in-depth quantitative analysis reveals the reasons behind our method's superior performance over DPO and showcases its versatility to other DPO variants. We release our repository at https://github.com/YJiangcm/BMC.","url_abs":"https://arxiv.org/abs/2408.07471v2","url_pdf":"https://arxiv.org/pdf/2408.07471v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bridging-and-modeling-correlations-in","repo_url":"https://github.com/YJiangcm/BMC","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"informativeness","task_name":"Informativeness"},{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"math","task_name":"Math"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"dpo","method_name":"DPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2408.07471","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.07471"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/YJiangcm/BMC","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"official":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ff720f8898acef89","entry":"call_llm_completion","repo":"YJiangcm/BMC","repo_kind":"official","path":"eval/QA/ARC/eval_coh.py","file_url":"https://github.com/YJiangcm/BMC/blob/HEAD/eval/QA/ARC/eval_coh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff720f8898acef89"}},{"code_sha256_prefix":"1525d8bc0fc8d5a0","entry":"generate_completions","repo":"YJiangcm/BMC","repo_kind":"official","path":"eval/math/eval/model_utils.py","file_url":"https://github.com/YJiangcm/BMC/blob/HEAD/eval/math/eval/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1525d8bc0fc8d5a0"}},{"code_sha256_prefix":"7dfbedc27279839f","entry":"get_json_list","repo":"YJiangcm/BMC","repo_kind":"official","path":"eval/alpacaeval2/alpaca_eval_inference.py","file_url":"https://github.com/YJiangcm/BMC/blob/HEAD/eval/alpacaeval2/alpaca_eval_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7dfbedc27279839f"}},{"code_sha256_prefix":"d18bb47e880446f4","entry":"get_json_list2","repo":"YJiangcm/BMC","repo_kind":"official","path":"eval/arenahard/arena_hard_inference.py","file_url":"https://github.com/YJiangcm/BMC/blob/HEAD/eval/arenahard/arena_hard_inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d18bb47e880446f4"}},{"code_sha256_prefix":"fc47045d127ed8bc","entry":"load_demo","repo":"YJiangcm/BMC","repo_kind":"official","path":"eval/QA/ARC/eval_coh.py","file_url":"https://github.com/YJiangcm/BMC/blob/HEAD/eval/QA/ARC/eval_coh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc47045d127ed8bc"}},{"code_sha256_prefix":"8281e63b913995ed","entry":"load_jsonl_data","repo":"YJiangcm/BMC","repo_kind":"official","path":"eval/QA/ARC/eval_coh.py","file_url":"https://github.com/YJiangcm/BMC/blob/HEAD/eval/QA/ARC/eval_coh.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8281e63b913995ed"}},{"code_sha256_prefix":"c35f14527d73a1ed","entry":"shuffle","repo":"YJiangcm/BMC","repo_kind":"official","path":"eval/math/prepare_train_data.py","file_url":"https://github.com/YJiangcm/BMC/blob/HEAD/eval/math/prepare_train_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c35f14527d73a1ed"}},{"code_sha256_prefix":"ee71a94d9dd1b8e9","entry":"load_hf_lm_and_tokenizer","repo":"YJiangcm/BMC","repo_kind":"official","path":"eval/math/eval/model_utils.py","file_url":"https://github.com/YJiangcm/BMC/blob/HEAD/eval/math/eval/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee71a94d9dd1b8e9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}