{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-hierarchical-multi-step-reward-models","title":"Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models","arxiv_id":"2503.13551","date":"2025-03-16","proceeding":null,"authors":["Teng Wang","Zhangyi Jiang","Zhenqi He","Shenyang Tong","Wenhan Yang","Yanan Zheng","Zeyu Li","Zifan He","Hailei Gong"],"abstract":"Recent studies show that Large Language Models (LLMs) achieve strong reasoning capabilities through supervised fine-tuning or reinforcement learning. However, a key approach, the Process Reward Model (PRM), suffers from reward hacking, making it unreliable in identifying the best intermediate step. In addition, the cost of annotating reasoning processes for reward modeling is high, making large-scale collection of high-quality data challenging. To address this, we propose a novel reward model approach called the Hierarchical Reward Model (HRM), which evaluates both individual and consecutive reasoning steps at both fine-grained and coarse-grained levels. HRM excels at assessing multi-step reasoning coherence, especially when flawed steps are later corrected through self-reflection. To further reduce the cost of generating training data, we introduce a lightweight and effective data augmentation strategy called Hierarchical Node Compression (HNC), which merges two consecutive reasoning steps into one within the tree structure. By applying HNC to MCTS-generated reasoning trajectories, we enhance the diversity and robustness of HRM training data while introducing controlled noise with minimal computational overhead. Empirical results on the PRM800K dataset show that HRM, together with HNC, provides more stable and reliable evaluations than PRM. Furthermore, cross-domain evaluations on the MATH500 and GSM8K datasets demonstrate HRM's strong generalization and robustness across a variety of reasoning tasks.","url_abs":"https://arxiv.org/abs/2503.13551v3","url_pdf":"https://arxiv.org/pdf/2503.13551v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-hierarchical-multi-step-reward-models","repo_url":"https://github.com/tengwang0318/hierarchial_reward_model","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"gsm8k","task_name":"GSM8K"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2503.13551","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.13551"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tengwang0318/hierarchial_reward_model","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4d2a6bb303deec5a","entry":"load_all_pickle_file_paths","repo":"tengwang0318/hierarchial_reward_model","repo_kind":"official","path":"self-training/construct_hrm_train_data.py","file_url":"https://github.com/tengwang0318/hierarchial_reward_model/blob/HEAD/self-training/construct_hrm_train_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4d2a6bb303deec5a"}},{"code_sha256_prefix":"b1a842cfb67e9b57","entry":"preprocess_function","repo":"tengwang0318/hierarchial_reward_model","repo_kind":"official","path":"self-training/choose_best_reward_model.py","file_url":"https://github.com/tengwang0318/hierarchial_reward_model/blob/HEAD/self-training/choose_best_reward_model.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1a842cfb67e9b57"}},{"code_sha256_prefix":"4724cec414f02f5b","entry":"replace_string","repo":"tengwang0318/hierarchial_reward_model","repo_kind":"official","path":"self-training/construct_hrm_train_data.py","file_url":"https://github.com/tengwang0318/hierarchial_reward_model/blob/HEAD/self-training/construct_hrm_train_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4724cec414f02f5b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}