{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-laws-for-reward-model-1","title":"Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms","arxiv_id":"2406.02900","date":"2024-06-05","proceeding":null,"authors":["Rafael Rafailov","Yaswanth Chittepu","Ryan Park","Harshit Sikchi","Joey Hejna","Bradley Knox","Chelsea Finn","Scott Niekum"],"abstract":"Reinforcement Learning from Human Feedback (RLHF) has been crucial to the recent success of Large Language Models (LLMs), however, it is often a complex and brittle process. In the classical RLHF framework, a reward model is first trained to represent human preferences, which is in turn used by an online reinforcement learning (RL) algorithm to optimize the LLM. A prominent issue with such methods is reward over-optimization or reward hacking, where performance as measured by the learned proxy reward model increases, but true quality plateaus or even deteriorates. Direct Alignment Algorithms (DDAs) like Direct Preference Optimization have emerged as alternatives to the classical RLHF pipeline by circumventing the reward modeling phase. However, although DAAs do not use a separate proxy reward model, they still commonly deteriorate from over-optimization. While the so-called reward hacking phenomenon is not well-defined for DAAs, we still uncover similar trends: at higher KL budgets, DAA algorithms exhibit similar degradation patterns to their classic RLHF counterparts. In particular, we find that DAA methods deteriorate not only across a wide range of KL budgets but also often before even a single epoch of the dataset is completed. Through extensive empirical experimentation, this work formulates and formalizes the reward over-optimization or hacking problem for DAAs and explores its consequences across objectives, training regimes, and model scales.","url_abs":"https://arxiv.org/abs/2406.02900v2","url_pdf":"https://arxiv.org/pdf/2406.02900v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2406.02900","atlas_url":"https://app.syntology.ai/?focus=2406.02900","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.02900"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/eric-mitchell/direct-preference-optimization","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/openai/summarize-from-feedback","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran_draft_wrong":2,"ran":8},"by_repo_kind":{"found_in_text":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/models/ops.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/models/ops.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"38486d3cdfec9c7f","entry":"avg_negative","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/sample.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/sample.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"38486d3cdfec9c7f"}},{"code_sha256_prefix":"4b3f35a4328b7f8a","entry":"dim_to_shard","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/query_response_model.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/query_response_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4b3f35a4328b7f8a"}},{"code_sha256_prefix":"8d23fbe2b99b840b","entry":"gelu","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/models/ops.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/models/ops.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8d23fbe2b99b840b"}},{"code_sha256_prefix":"930531cadabf9316","entry":"nucleus_sampler","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/models/sample_fns.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/models/sample_fns.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"930531cadabf9316"}},{"code_sha256_prefix":"729302b73bcdbc1f","entry":"parameter_name_to_sharding_dim","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/query_response_model.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/query_response_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"729302b73bcdbc1f"}},{"code_sha256_prefix":"e21f72f1a9834aef","entry":"quick_gelu","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/models/ops.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/models/ops.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e21f72f1a9834aef"}},{"code_sha256_prefix":"b3248a4cb21140cd","entry":"softmax_xent_loss_fn","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/models/loss_functions.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/models/loss_functions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b3248a4cb21140cd"}},{"code_sha256_prefix":"1dce5b82465df17b","entry":"standard","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/models/sample_fns.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/models/sample_fns.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"1dce5b82465df17b"}},{"code_sha256_prefix":"7295fd34fe7e12f9","entry":"torch_loader","repo":"openai/summarize-from-feedback","repo_kind":"found_in_text","path":"summarize_from_feedback/task_data.py","file_url":"https://github.com/openai/summarize-from-feedback/blob/HEAD/summarize_from_feedback/task_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7295fd34fe7e12f9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}