{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vanishing-gradients-in-reinforcement","title":"Vanishing Gradients in Reinforcement Finetuning of Language Models","arxiv_id":"2310.20703","date":"2023-10-31","proceeding":null,"authors":["Noam Razin","Hattie Zhou","Omid Saremi","Vimal Thilak","Arwen Bradley","Preetum Nakkiran","Joshua Susskind","Etai Littwin"],"abstract":"Pretrained language models are commonly aligned with human preferences and downstream tasks via reinforcement finetuning (RFT), which refers to maximizing a (possibly learned) reward function using policy gradient algorithms. This work identifies a fundamental optimization obstacle in RFT: we prove that the expected gradient for an input vanishes when its reward standard deviation under the model is small, even if the expected reward is far from optimal. Through experiments on an RFT benchmark and controlled environments, as well as a theoretical analysis, we then demonstrate that vanishing gradients due to small reward standard deviation are prevalent and detrimental, leading to extremely slow reward maximization. Lastly, we explore ways to overcome vanishing gradients in RFT. We find the common practice of an initial supervised finetuning (SFT) phase to be the most promising candidate, which sheds light on its importance in an RFT pipeline. Moreover, we show that a relatively small number of SFT optimization steps on as few as 1% of the input samples can suffice, indicating that the initial SFT phase need not be expensive in terms of compute and data labeling efforts. Overall, our results emphasize that being mindful for inputs whose expected gradient vanishes, as measured by the reward standard deviation, is crucial for successful execution of RFT.","url_abs":"https://arxiv.org/abs/2310.20703v3","url_pdf":"https://arxiv.org/pdf/2310.20703v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vanishing-gradients-in-reinforcement","repo_url":"https://github.com/apple/ml-rlgrad","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[],"methods":[{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.20703","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.20703"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/apple/ml-rlgrad","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":1,"unverified":5},"by_repo_kind":{"official":{"samples":6,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"273737207da9fb73","entry":"get_max_prompt_length","repo":"apple/ml-rlgrad","repo_kind":"official","path":"rl4lms_extract_samples_from_model_runner.py","file_url":"https://github.com/apple/ml-rlgrad/blob/HEAD/rl4lms_extract_samples_from_model_runner.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"273737207da9fb73"}},{"code_sha256_prefix":"c59cb856dad5f089","entry":"encode_example","repo":"apple/ml-rlgrad","repo_kind":"official","path":"bleurt/encoding.py","file_url":"https://github.com/apple/ml-rlgrad/blob/HEAD/bleurt/encoding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c59cb856dad5f089"}},{"code_sha256_prefix":"928f493bec0ffc69","entry":"ranking_loss","repo":"apple/ml-rlgrad","repo_kind":"official","path":"bleurt/model.py","file_url":"https://github.com/apple/ml-rlgrad/blob/HEAD/bleurt/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"928f493bec0ffc69"}},{"code_sha256_prefix":"f57743217d50baa3","entry":"read_bleurt_config","repo":"apple/ml-rlgrad","repo_kind":"official","path":"bleurt/checkpoint.py","file_url":"https://github.com/apple/ml-rlgrad/blob/HEAD/bleurt/checkpoint.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f57743217d50baa3"}},{"code_sha256_prefix":"f71891eba726ed24","entry":"serialize_example","repo":"apple/ml-rlgrad","repo_kind":"official","path":"bleurt/encoding.py","file_url":"https://github.com/apple/ml-rlgrad/blob/HEAD/bleurt/encoding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f71891eba726ed24"}},{"code_sha256_prefix":"74a38d5d8b44259d","entry":"serialize_raw_example","repo":"apple/ml-rlgrad","repo_kind":"official","path":"bleurt/encoding.py","file_url":"https://github.com/apple/ml-rlgrad/blob/HEAD/bleurt/encoding.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"74a38d5d8b44259d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}