{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/weighted-reward-preference-optimization-for","title":"Weighted-Reward Preference Optimization for Implicit Model Fusion","arxiv_id":"2412.03187","date":"2024-12-04","proceeding":null,"authors":["ZiYi Yang","Fanqi Wan","Longguang Zhong","Tianyuan Shi","Xiaojun Quan"],"abstract":"While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against GPT-4-0314 on Arena-Hard. Our code is available at \\url{https://github.com/SLIT-AI/WRPO}.","url_abs":"https://arxiv.org/abs/2412.03187v1","url_pdf":"https://arxiv.org/pdf/2412.03187v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"weighted-reward-preference-optimization-for","repo_url":"https://github.com/SLIT-AI/WRPO","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"weighted-reward-preference-optimization-for","repo_url":"https://github.com/fanqiwan/fuseai","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"weighted-reward-preference-optimization-for","repo_url":"https://github.com/fanqiwan/fusellm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"weighted-reward-preference-optimization-for","repo_url":"https://github.com/slit-ai/fusechat-3.0","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2412.03187","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.03187"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/SLIT-AI/WRPO","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/slit-ai/fusechat-3.0","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fanqiwan/fuseai","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fanqiwan/fusellm","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":1,"ran_violates":1,"unverified":5},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":13,"ran":8,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":14,"samples":[{"code_sha256_prefix":"524b27f03377a248","entry":"check_nan","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseChat/train/filter_nan.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseChat/train/filter_nan.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"524b27f03377a248"}},{"code_sha256_prefix":"fd32354386c6ce83","entry":"check_nan_base","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseChat/train/filter_nan.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseChat/train/filter_nan.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd32354386c6ce83"}},{"code_sha256_prefix":"98b9e3b2ff2e30b8","entry":"dict_to_list","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseChat/train/align_token_and_vocab.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseChat/train/align_token_and_vocab.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"98b9e3b2ff2e30b8"}},{"code_sha256_prefix":"1525d8bc0fc8d5a0","entry":"generate_completions","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseO1-Preview/math_evaluation/model_utils.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseO1-Preview/math_evaluation/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1525d8bc0fc8d5a0"}},{"code_sha256_prefix":"1082508e65f77331","entry":"get_logger","repo":"fanqiwan/fusellm","repo_kind":"listed","path":"FuseLLM/src/utils/others.py","file_url":"https://github.com/fanqiwan/fusellm/blob/HEAD/FuseLLM/src/utils/others.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1082508e65f77331"}},{"code_sha256_prefix":"8e08bca34751633c","entry":"list_to_dict","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseChat/train/align_token_and_vocab.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseChat/train/align_token_and_vocab.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8e08bca34751633c"}},{"code_sha256_prefix":"030b2d5e6e147a1d","entry":"pad_to_length","repo":"slit-ai/wrpo","repo_kind":"official","path":"scripts/wrpo_trainer.py","file_url":"https://github.com/slit-ai/wrpo/blob/HEAD/scripts/wrpo_trainer.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"030b2d5e6e147a1d"}},{"code_sha256_prefix":"0bf414a2b44babe6","entry":"raise_warning_for_incompatible_cpu_offloading_configuration","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseChat/train/model/model_adapter.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseChat/train/model/model_adapter.py","link_basis":"plan_row","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bf414a2b44babe6"}},{"code_sha256_prefix":"d0b132e4577e5bbb","entry":"sigmoid","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseChat/train/align_token_and_vocab.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseChat/train/align_token_and_vocab.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d0b132e4577e5bbb"}},{"code_sha256_prefix":"df472be23644090f","entry":"get_base_model","repo":"fanqiwan/fusellm","repo_kind":"listed","path":"FuseLLM/src/utils/others.py","file_url":"https://github.com/fanqiwan/fusellm/blob/HEAD/FuseLLM/src/utils/others.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df472be23644090f"}},{"code_sha256_prefix":"502fd3534e0509c1","entry":"get_conv_template","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseChat/train/conversation.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseChat/train/conversation.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"502fd3534e0509c1"}},{"code_sha256_prefix":"4bec2455307c11b7","entry":"get_gpu_memory","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseChat/train/model/model_adapter.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseChat/train/model/model_adapter.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bec2455307c11b7"}},{"code_sha256_prefix":"503f70e78560aa9e","entry":"get_tokenizer","repo":"fanqiwan/fusellm","repo_kind":"listed","path":"FuseLLM/src/utils/others.py","file_url":"https://github.com/fanqiwan/fusellm/blob/HEAD/FuseLLM/src/utils/others.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"503f70e78560aa9e"}},{"code_sha256_prefix":"ee71a94d9dd1b8e9","entry":"load_hf_lm_and_tokenizer","repo":"fanqiwan/fuseai","repo_kind":"listed","path":"FuseO1-Preview/math_evaluation/model_utils.py","file_url":"https://github.com/fanqiwan/fuseai/blob/HEAD/FuseO1-Preview/math_evaluation/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee71a94d9dd1b8e9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}