{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/discriminative-policy-optimization-for-token","title":"Discriminative Policy Optimization for Token-Level Reward Models","arxiv_id":"2505.23363","date":"2025-05-29","proceeding":null,"authors":["Hongzhan Chen","Tao Yang","Shiping Gao","Ruijun Chen","Xiaojun Quan","Hongtao Tian","Ting Yao"],"abstract":"Process reward models (PRMs) provide more nuanced supervision compared to outcome reward models (ORMs) for optimizing policy models, positioning them as a promising approach to enhancing the capabilities of LLMs in complex reasoning tasks. Recent efforts have advanced PRMs from step-level to token-level granularity by integrating reward modeling into the training of generative models, with reward scores derived from token generation probabilities. However, the conflict between generative language modeling and reward modeling may introduce instability and lead to inaccurate credit assignments. To address this challenge, we revisit token-level reward assignment by decoupling reward modeling from language generation and derive a token-level reward model through the optimization of a discriminative policy, termed the Q-function Reward Model (Q-RM). We theoretically demonstrate that Q-RM explicitly learns token-level Q-functions from preference data without relying on fine-grained annotations. In our experiments, Q-RM consistently outperforms all baseline methods across various benchmarks. For example, when integrated into PPO/REINFORCE algorithms, Q-RM enhances the average Pass@1 score by 5.85/4.70 points on mathematical reasoning tasks compared to the ORM baseline, and by 4.56/5.73 points compared to the token-level PRM counterpart. Moreover, reinforcement learning with Q-RM significantly enhances training efficiency, achieving convergence 12 times faster than ORM on GSM8K and 11 times faster than step-level PRM on MATH. Code and data are available at https://github.com/homzer/Q-RM.","url_abs":"https://arxiv.org/abs/2505.23363v1","url_pdf":"https://arxiv.org/pdf/2505.23363v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"discriminative-policy-optimization-for-token","repo_url":"https://github.com/homzer/q-rm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"gsm8k","task_name":"GSM8K"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"},{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.23363","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.23363"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/homzer/Q-RM","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/homzer/q-rm","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"9aca49f69144329d","entry":"DPOLoss","repo":"homzer/q-rm","repo_kind":"official","path":"src/rewards/strategy.py","file_url":"https://github.com/homzer/q-rm/blob/HEAD/src/rewards/strategy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9aca49f69144329d"}},{"code_sha256_prefix":"ae4c4703ba50654a","entry":"Loss","repo":"homzer/q-rm","repo_kind":"official","path":"src/rewards/strategy.py","file_url":"https://github.com/homzer/q-rm/blob/HEAD/src/rewards/strategy.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ae4c4703ba50654a"}},{"code_sha256_prefix":"fb3698bb90fe853f","entry":"ImplicitPRMLoss","repo":"homzer/q-rm","repo_kind":"official","path":"src/rewards/strategy.py","file_url":"https://github.com/homzer/q-rm/blob/HEAD/src/rewards/strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fb3698bb90fe853f"}},{"code_sha256_prefix":"66e399ccbbbf9822","entry":"PointwiseVerifierStrategy","repo":"homzer/q-rm","repo_kind":"official","path":"src/rewards/strategy.py","file_url":"https://github.com/homzer/q-rm/blob/HEAD/src/rewards/strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"66e399ccbbbf9822"}},{"code_sha256_prefix":"baed4961ca4cf190","entry":"PointwiseVerifierStrategyForImplicitPRM","repo":"homzer/q-rm","repo_kind":"official","path":"src/rewards/strategy.py","file_url":"https://github.com/homzer/q-rm/blob/HEAD/src/rewards/strategy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"baed4961ca4cf190"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}