{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/confronting-reward-model-overoptimization","title":"Confronting Reward Model Overoptimization with Constrained RLHF","arxiv_id":"2310.04373","date":"2023-10-06","proceeding":null,"authors":["Ted Moskovitz","Aaditya K. Singh","DJ Strouse","Tuomas Sandholm","Ruslan Salakhutdinov","Anca D. Dragan","Stephen Mcaleer"],"abstract":"Large language models are typically aligned with human preferences by optimizing $\\textit{reward models}$ (RMs) fitted to human feedback. However, human preferences are multi-faceted, and it is increasingly common to derive reward from a composition of simpler reward models which each capture a different aspect of language quality. This itself presents a challenge, as it is difficult to appropriately weight these component RMs when combining them. Compounding this difficulty, because any RM is only a proxy for human evaluation, this process is vulnerable to $\\textit{overoptimization}$, wherein past a certain point, accumulating higher reward is associated with worse human ratings. In this paper, we perform, to our knowledge, the first study on overoptimization in composite RMs, showing that correlation between component RMs has a significant effect on the locations of these points. We then introduce an approach to solve this issue using constrained reinforcement learning as a means of preventing the agent from exceeding each RM's threshold of usefulness. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally expressed by Lagrange multipliers. As a result, each RM stays within the range at which it is an effective proxy, improving evaluation performance. Finally, we introduce an adaptive method using gradient-free optimization to identify and optimize towards these points during a single run.","url_abs":"https://arxiv.org/abs/2310.04373v2","url_pdf":"https://arxiv.org/pdf/2310.04373v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"confronting-reward-model-overoptimization","repo_url":"https://github.com/tedmoskovitz/constrainedrl4lms","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"model","task_name":"model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.04373","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.04373"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tedmoskovitz/constrainedrl4lms","reach":{"status":"ok"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"92e1118e4e5f774c","entry":"conjugate_gradient_solver","repo":"tedmoskovitz/constrainedrl4lms","repo_kind":"official","path":"rl4lms/algorithms/common/algo_utils.py","file_url":"https://github.com/tedmoskovitz/constrainedrl4lms/blob/HEAD/rl4lms/algorithms/common/algo_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"92e1118e4e5f774c"}},{"code_sha256_prefix":"91200094734b3131","entry":"find_products","repo":"tedmoskovitz/constrainedrl4lms","repo_kind":"official","path":"rl4lms/core_components/sweep.py","file_url":"https://github.com/tedmoskovitz/constrainedrl4lms/blob/HEAD/rl4lms/core_components/sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"91200094734b3131"}},{"code_sha256_prefix":"206123801663c69f","entry":"flat_grad","repo":"tedmoskovitz/constrainedrl4lms","repo_kind":"official","path":"rl4lms/algorithms/common/algo_utils.py","file_url":"https://github.com/tedmoskovitz/constrainedrl4lms/blob/HEAD/rl4lms/algorithms/common/algo_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"206123801663c69f"}},{"code_sha256_prefix":"e38b5971551fdee2","entry":"get_dict_obj","repo":"tedmoskovitz/constrainedrl4lms","repo_kind":"official","path":"rl4lms/core_components/sweep.py","file_url":"https://github.com/tedmoskovitz/constrainedrl4lms/blob/HEAD/rl4lms/core_components/sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e38b5971551fdee2"}},{"code_sha256_prefix":"821b0061bc694ddf","entry":"quantile_huber_loss","repo":"tedmoskovitz/constrainedrl4lms","repo_kind":"official","path":"rl4lms/algorithms/common/algo_utils.py","file_url":"https://github.com/tedmoskovitz/constrainedrl4lms/blob/HEAD/rl4lms/algorithms/common/algo_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"821b0061bc694ddf"}},{"code_sha256_prefix":"b41d462e09610b1c","entry":"to_expand","repo":"tedmoskovitz/constrainedrl4lms","repo_kind":"official","path":"rl4lms/core_components/sweep.py","file_url":"https://github.com/tedmoskovitz/constrainedrl4lms/blob/HEAD/rl4lms/core_components/sweep.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b41d462e09610b1c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}