{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mutual-taught-for-co-adapting-policy-and","title":"Mutual-Taught for Co-adapting Policy and Reward Models","arxiv_id":"2506.06292","date":"2025-05-17","proceeding":null,"authors":["Tianyuan Shi","Canbin Huang","Fanqi Wan","Longguang Zhong","ZiYi Yang","Weizhou Shen","Xiaojun Quan","Ming Yan"],"abstract":"During the preference optimization of large language models (LLMs), distribution shifts may arise between newly generated model samples and the data used to train the reward model (RM). This shift reduces the efficacy of the RM, which in turn negatively impacts the performance of the policy model (PM). To address this challenge, we propose Mutual-Taught, a self-training method that iteratively improves both the PM and RM without requiring additional human annotation. Our approach mirrors the expectation-maximization (EM) algorithm. In the E-step, the PM is updated using feedback from the current RM, guiding the PM toward a better approximation of the latent optimal preference distribution. In the M-step, we update the RM by constructing training data from the outputs of the PM before and after the E-step update. This process ensures that the RM adapts to the evolving policy distribution. Experimental results demonstrate that this iterative approach leads to consistent improvements in both models. Specifically, our 8B policy model, LLaMA-3-8B-Instruct-MT, achieves a length-controlled win rate of 54.1\\% on AlpacaEval-2, while our 8B reward model, FsfairX-LLaMA3-RM-MT, performs on par with GPT-4o-2024-08-06 on RewardBench.","url_abs":"https://arxiv.org/abs/2506.06292v2","url_pdf":"https://arxiv.org/pdf/2506.06292v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2506.06292","atlas_url":"https://app.syntology.ai/?focus=2506.06292","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.06292"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/Stycoo/Mutual-Taught","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":1,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":7,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"10ff293597eab810","entry":"maybe_torch","repo":"Stycoo/Mutual-Taught","repo_kind":"found_in_text","path":"reward_model_merge/rm_merge_custom.py","file_url":"https://github.com/Stycoo/Mutual-Taught/blob/HEAD/reward_model_merge/rm_merge_custom.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"10ff293597eab810"}},{"code_sha256_prefix":"f7d5662f1399084a","entry":"build_dpo_dataset","repo":"Stycoo/Mutual-Taught","repo_kind":"found_in_text","path":"completion_inference/build_dataset.py","file_url":"https://github.com/Stycoo/Mutual-Taught/blob/HEAD/completion_inference/build_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f7d5662f1399084a"}},{"code_sha256_prefix":"11ae53d561cabab0","entry":"create_node","repo":"Stycoo/Mutual-Taught","repo_kind":"found_in_text","path":"completion_inference/completion_infer.py","file_url":"https://github.com/Stycoo/Mutual-Taught/blob/HEAD/completion_inference/completion_infer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11ae53d561cabab0"}},{"code_sha256_prefix":"ed054b3a2f7dde35","entry":"lerp","repo":"Stycoo/Mutual-Taught","repo_kind":"found_in_text","path":"reward_model_merge/rm_merge_custom.py","file_url":"https://github.com/Stycoo/Mutual-Taught/blob/HEAD/reward_model_merge/rm_merge_custom.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed054b3a2f7dde35"}},{"code_sha256_prefix":"1acc67cef16f4df5","entry":"read_json_file","repo":"Stycoo/Mutual-Taught","repo_kind":"found_in_text","path":"completion_inference/build_dataset.py","file_url":"https://github.com/Stycoo/Mutual-Taught/blob/HEAD/completion_inference/build_dataset.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1acc67cef16f4df5"}},{"code_sha256_prefix":"343787d4eb1da770","entry":"slerp","repo":"Stycoo/Mutual-Taught","repo_kind":"found_in_text","path":"reward_model_merge/rm_merge_custom.py","file_url":"https://github.com/Stycoo/Mutual-Taught/blob/HEAD/reward_model_merge/rm_merge_custom.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"343787d4eb1da770"}},{"code_sha256_prefix":"615a32d374e639ea","entry":"split_by_newlines","repo":"Stycoo/Mutual-Taught","repo_kind":"found_in_text","path":"completion_inference/completion_infer.py","file_url":"https://github.com/Stycoo/Mutual-Taught/blob/HEAD/completion_inference/completion_infer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"615a32d374e639ea"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}