{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2601-19404","title":"RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimization","arxiv_id":"2601.19404","date":"2026-01-27","proceeding":null,"authors":["Hongzhu Yi","Xinming Wang","Zhenghao zhang","Tianyu Zong","Yuanxiang Wang","Jun Xie","Tao Yu","Haopeng Jin","Kaixin Xu","Feng Chen","Jiahuan Chen","Yujia Yang","Zhenyu Guan","Bingkang Shi","Jungang Xu"],"abstract":"Within the domain of large language models, reinforcement fine-tuning algorithms necessitate the generation of a complete reasoning trajectory beginning from the input query, which incurs significant computational overhead during the rollout phase of training. To address this issue, we analyze the impact of different segments of the reasoning path on the correctness of the final result and, based on these insights, propose Reinforcement Fine-Tuning with Partial Reasoning Optimization (RPO), a plug-and-play reinforcement fine-tuning algorithm. Unlike traditional reinforcement fine-tuning algorithms that generate full reasoning paths, RPO trains the model by generating suffixes of the reasoning path using experience cache. During the rollout phase of training, RPO reduces token generation in this phase by approximately 95%, greatly lowering the theoretical time overhead. Compared with full-path reinforcement fine-tuning algorithms, RPO reduces the training time of the 1.5B model by 90% and the 7B model by 72%. At the same time, it can be integrated with typical algorithms such as GRPO and DAPO, enabling them to achieve training acceleration while maintaining performance comparable to the original algorithms. Our code is open-sourced at https://github.com/yhz5613813/RPO.","url_abs":"https://arxiv.org/abs/2601.19404","url_pdf":"https://arxiv.org/pdf/2601.19404","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2601.19404","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2601.19404"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/yhz5613813/RPO","reach":null}],"summary":{"ran":1,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":5,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9b92c44e1bde6319","entry":"get_callbacks","repo":"yhz5613813/RPO","repo_kind":"found_in_text","path":"src/open_r1/utils/callbacks.py","file_url":"https://github.com/yhz5613813/RPO/blob/HEAD/src/open_r1/utils/callbacks.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b92c44e1bde6319"}},{"code_sha256_prefix":"0a5c0b6fb53cd48c","entry":"format_reward","repo":"yhz5613813/RPO","repo_kind":"found_in_text","path":"src/open_r1/rewards.py","file_url":"https://github.com/yhz5613813/RPO/blob/HEAD/src/open_r1/rewards.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a5c0b6fb53cd48c"}},{"code_sha256_prefix":"18c3de9b6f483fea","entry":"make_conversation","repo":"yhz5613813/RPO","repo_kind":"found_in_text","path":"export_data.py","file_url":"https://github.com/yhz5613813/RPO/blob/HEAD/export_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"18c3de9b6f483fea"}},{"code_sha256_prefix":"60875b0a0cc5a189","entry":"pad_tensor_list","repo":"yhz5613813/RPO","repo_kind":"found_in_text","path":"export_data.py","file_url":"https://github.com/yhz5613813/RPO/blob/HEAD/export_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"60875b0a0cc5a189"}},{"code_sha256_prefix":"f69ad4ba6703f439","entry":"process_final_data","repo":"yhz5613813/RPO","repo_kind":"found_in_text","path":"export_data.py","file_url":"https://github.com/yhz5613813/RPO/blob/HEAD/export_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f69ad4ba6703f439"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}