{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/process-reinforcement-through-implicit","title":"Process Reinforcement through Implicit Rewards","arxiv_id":"2502.01456","date":"2025-02-03","proceeding":null,"authors":["Ganqu Cui","Lifan Yuan","Zefan Wang","Hanbin Wang","Wendi Li","Bingxiang He","Yuchen Fan","Tianyu Yu","Qixin Xu","Weize Chen","Jiarui Yuan","Huayu Chen","Kaiyan Zhang","Xingtai Lv","Shuo Wang","Yuan YAO","Xu Han","Hao Peng","Yu Cheng","Zhiyuan Liu","Maosong Sun","BoWen Zhou","Ning Ding"],"abstract":"Dense process rewards have proven a more effective alternative to the sparse outcome-level rewards in the inference-time scaling of large language models (LLMs), particularly in tasks requiring complex multi-step reasoning. While dense rewards also offer an appealing choice for the reinforcement learning (RL) of LLMs since their fine-grained rewards have the potential to address some inherent issues of outcome rewards, such as training efficiency and credit assignment, this potential remains largely unrealized. This can be primarily attributed to the challenges of training process reward models (PRMs) online, where collecting high-quality process labels is prohibitively expensive, making them particularly vulnerable to reward hacking. To address these challenges, we propose PRIME (Process Reinforcement through IMplicit rEwards), which enables online PRM updates using only policy rollouts and outcome labels through implict process rewards. PRIME combines well with various advantage functions and forgoes the dedicated reward model training phrase that existing approaches require, substantially reducing the development overhead. We demonstrate PRIME's effectiveness on competitional math and coding. Starting from Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several key reasoning benchmarks over the SFT model. Notably, our resulting model, Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning benchmarks with 10% of its training data.","url_abs":"https://arxiv.org/abs/2502.01456v1","url_pdf":"https://arxiv.org/pdf/2502.01456v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"process-reinforcement-through-implicit","repo_url":"https://github.com/prime-rl/prime","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"process-reinforcement-through-implicit","repo_url":"https://github.com/OpenLLMAI/OpenLLaMA2","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"process-reinforcement-through-implicit","repo_url":"https://github.com/OpenLLMAI/OpenRLHF","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"process-reinforcement-through-implicit","repo_url":"https://github.com/cjreinforce/pure","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"process-reinforcement-through-implicit","repo_url":"https://github.com/openrlhf/openrlhf","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"math","task_name":"Math"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.01456","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.01456"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OpenLLMAI/OpenRLHF","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/prime-rl/prime","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/openrlhf/openrlhf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cjreinforce/pure","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/OpenLLMAI/OpenLLaMA2","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":1,"ran":1,"unverified":15},"by_repo_kind":{"official":{"samples":17,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e080db2eaa81fdb3","entry":"estimate_pass_at_k","repo":"prime-rl/prime","repo_kind":"official","path":"eval/utils/evaluation.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/eval/utils/evaluation.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e080db2eaa81fdb3"}},{"code_sha256_prefix":"101b1e54fcdadf90","entry":"normalize_answer","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/math_util/math_normalize.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/math_util/math_normalize.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"101b1e54fcdadf90"}},{"code_sha256_prefix":"4af93356397e2583","entry":"create_filter_prompt","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/stage2_format_choice.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/stage2_format_choice.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4af93356397e2583"}},{"code_sha256_prefix":"c7721490607b846e","entry":"create_question_prompt","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/stage2_format_choice.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/stage2_format_choice.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c7721490607b846e"}},{"code_sha256_prefix":"120a553951e00af6","entry":"create_solution_prompt","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/stage2_format_choice.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/stage2_format_choice.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"120a553951e00af6"}},{"code_sha256_prefix":"bb175bff1060d928","entry":"handle_base","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/math_util/grader.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/math_util/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb175bff1060d928"}},{"code_sha256_prefix":"2a180d3156997924","entry":"is_digit","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/math_util/grader.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/math_util/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2a180d3156997924"}},{"code_sha256_prefix":"960059cee2753e3f","entry":"normalize","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/math_util/grader.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/math_util/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"960059cee2753e3f"}},{"code_sha256_prefix":"2f2aa5fc73311d4e","entry":"parallel_evaluate_continual_async","repo":"prime-rl/prime","repo_kind":"official","path":"training/verl/utils/reward_score/prime.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/training/verl/utils/reward_score/prime.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f2aa5fc73311d4e"}},{"code_sha256_prefix":"aacfe469bd668c58","entry":"process_completion","repo":"prime-rl/prime","repo_kind":"official","path":"training/verl/utils/reward_score/prime.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/training/verl/utils/reward_score/prime.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aacfe469bd668c58"}},{"code_sha256_prefix":"86bc2b52fc4de062","entry":"process_fn","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/stage1_filter.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/stage1_filter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"86bc2b52fc4de062"}},{"code_sha256_prefix":"4dc6b4c66100580d","entry":"process_fn","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/stage3_merge.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/stage3_merge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4dc6b4c66100580d"}},{"code_sha256_prefix":"af48c07b88855b2d","entry":"process_humaneval_test","repo":"prime-rl/prime","repo_kind":"official","path":"eval/utils/evaluation_leetcode.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/eval/utils/evaluation_leetcode.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af48c07b88855b2d"}},{"code_sha256_prefix":"38c21e0e3a1e76d2","entry":"process_row_with_timeout","repo":"prime-rl/prime","repo_kind":"official","path":"training/verl/utils/reward_score/prime.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/training/verl/utils/reward_score/prime.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38c21e0e3a1e76d2"}},{"code_sha256_prefix":"562c0ef13357b53b","entry":"read_problems","repo":"prime-rl/prime","repo_kind":"official","path":"eval/utils/data.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/eval/utils/data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"562c0ef13357b53b"}},{"code_sha256_prefix":"860555756a95114b","entry":"tell_choice","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/stage1_filter.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/stage1_filter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"860555756a95114b"}},{"code_sha256_prefix":"652e9a2cd5d2bd69","entry":"tell_choice","repo":"prime-rl/prime","repo_kind":"official","path":"data_preprocessing/stage3_merge.py","file_url":"https://github.com/prime-rl/prime/blob/HEAD/data_preprocessing/stage3_merge.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"652e9a2cd5d2bd69"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}