{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reward-model-ensembles-help-mitigate","title":"Reward Model Ensembles Help Mitigate Overoptimization","arxiv_id":"2310.02743","date":"2023-10-04","proceeding":null,"authors":["Thomas Coste","Usman Anwar","Robert Kirk","David Krueger"],"abstract":"Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As part of this process, learned reward models are used to approximately model human preferences. However, as imperfect representations of the \"true\" reward, these learned reward models are susceptible to overoptimization. Gao et al. (2023) studied this phenomenon in a synthetic human feedback setup with a significantly larger \"gold\" reward model acting as the true reward (instead of humans) and showed that overoptimization remains a persistent problem regardless of the size of the proxy reward model and training data used. Using a similar setup, we conduct a systematic study to evaluate the efficacy of using ensemble-based conservative optimization objectives, specifically worst-case optimization (WCO) and uncertainty-weighted optimization (UWO), for mitigating reward model overoptimization when using two optimization methods: (a) best-of-n sampling (BoN) (b) proximal policy optimization (PPO). We additionally extend the setup of Gao et al. (2023) to include 25% label noise to better mirror real-world conditions. Both with and without label noise, we find that conservative optimization practically eliminates overoptimization and improves performance by up to 70% for BoN sampling. For PPO, ensemble-based conservative optimization always reduces overoptimization and outperforms single reward model optimization. Moreover, combining it with a small KL penalty successfully prevents overoptimization at no performance cost. Overall, our results demonstrate that ensemble-based conservative optimization can effectively counter overoptimization.","url_abs":"https://arxiv.org/abs/2310.02743v2","url_pdf":"https://arxiv.org/pdf/2310.02743v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reward-model-ensembles-help-mitigate","repo_url":"https://github.com/tlc4418/llm_optimization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"reward-model-ensembles-help-mitigate","repo_url":"https://github.com/AnamikaLochab/EBRM","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"model-optimization","task_name":"Model Optimization"},{"task_slug":"model","task_name":"model"}],"methods":[{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.02743","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.02743"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tlc4418/llm_optimization","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AnamikaLochab/EBRM","reach":null}],"summary":{"ran_draft_wrong":5},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"fc18eae85db60add","entry":"batch_loss_function","repo":"AnamikaLochab/EBRM","repo_kind":"listed","path":"src/reward_modeling/ebm_training/ebm_nce_plus.py","file_url":"https://github.com/AnamikaLochab/EBRM/blob/HEAD/src/reward_modeling/ebm_training/ebm_nce_plus.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fc18eae85db60add"}},{"code_sha256_prefix":"8bdebfd6cb172535","entry":"build_distributions","repo":"AnamikaLochab/EBRM","repo_kind":"listed","path":"src/reward_modeling/ebm_training/ebm_nce_plus.py","file_url":"https://github.com/AnamikaLochab/EBRM/blob/HEAD/src/reward_modeling/ebm_training/ebm_nce_plus.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bdebfd6cb172535"}},{"code_sha256_prefix":"46512a251e5a96f2","entry":"compute_log_Z_batch","repo":"AnamikaLochab/EBRM","repo_kind":"listed","path":"src/reward_modeling/ebm_training/ebm_nce_plus.py","file_url":"https://github.com/AnamikaLochab/EBRM/blob/HEAD/src/reward_modeling/ebm_training/ebm_nce_plus.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"46512a251e5a96f2"}},{"code_sha256_prefix":"9af050606d9b4cea","entry":"compute_metrics","repo":"tlc4418/llm_optimization","repo_kind":"official","path":"src/sft/trainer_sft.py","file_url":"https://github.com/tlc4418/llm_optimization/blob/HEAD/src/sft/trainer_sft.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9af050606d9b4cea"}},{"code_sha256_prefix":"0b66a8919774c46f","entry":"preprocess_logits_for_metrics","repo":"tlc4418/llm_optimization","repo_kind":"official","path":"src/sft/trainer_sft.py","file_url":"https://github.com/tlc4418/llm_optimization/blob/HEAD/src/sft/trainer_sft.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b66a8919774c46f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}