{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reinforcement-learning-for-reasoning-in-large","title":"Reinforcement Learning for Reasoning in Large Language Models with One Training Example","arxiv_id":"2504.20571","date":"2025-04-29","proceeding":null,"authors":["Yiping Wang","Qing Yang","Zhiyuan Zeng","Liliang Ren","Lucas Liu","Baolin Peng","Hao Cheng","Xuehai He","Kuan Wang","Jianfeng Gao","Weizhu Chen","Shuohang Wang","Simon Shaolei Du","Yelong Shen"],"abstract":"We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model performance on MATH500 from 36.0% to 73.6%, and improves the average performance across six common mathematical reasoning benchmarks from 17.6% to 35.7%. This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6%, average: 35.9%), which includes the aforementioned example. Similar substantial improvements are observed across various models (Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and PPO), and different math examples (many of which yield approximately 30% or greater improvement on MATH500 when employed as a single training example). In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-domain generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term post-saturation generalization. Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the \"grokking\" phenomenon. We also show the critical role of promoting exploration (e.g., by adding entropy loss with an appropriate coefficient) in 1-shot RLVR training. As a bonus, we observe that applying entropy loss alone, without any outcome reward, significantly enhances Qwen2.5-Math-1.5B's performance on MATH500 by 27.4%. These findings can inspire future work on RLVR data efficiency and encourage a re-examination of both recent progress and the underlying mechanisms in RLVR. Our code, model, and data are open source at https://github.com/ypwang61/One-Shot-RLVR","url_abs":"https://arxiv.org/abs/2504.20571v1","url_pdf":"https://arxiv.org/pdf/2504.20571v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reinforcement-learning-for-reasoning-in-large","repo_url":"https://github.com/ypwang61/one-shot-rlvr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"domain-generalization","task_name":"Domain Generalization"},{"task_slug":"math","task_name":"Math"},{"task_slug":"mathematical-reasoning","task_name":"Mathematical Reasoning"}],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.20571","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.20571"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ypwang61/one-shot-rlvr","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_violates":1,"ran_fixture":1,"ran_draft_wrong":1,"unverified":12},"by_repo_kind":{"official":{"samples":16,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1525d8bc0fc8d5a0","entry":"generate_completions","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/model_utils.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1525d8bc0fc8d5a0"}},{"code_sha256_prefix":"d338a8cb44aeb6b4","entry":"is_multi_choice","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/math_eval.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/math_eval.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d338a8cb44aeb6b4"}},{"code_sha256_prefix":"21331a58f93375e4","entry":"union_tensor_dict","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"verl/protocol.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"21331a58f93375e4"}},{"code_sha256_prefix":"25f0ea3f460f6ce1","entry":"unpad_dataproto","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"verl/protocol.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"25f0ea3f460f6ce1"}},{"code_sha256_prefix":"809c0178a7ae90ff","entry":"choice_answer_clean","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/grader.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"809c0178a7ae90ff"}},{"code_sha256_prefix":"81082f28e1440f9d","entry":"compare_numerical_ans","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/math_utils.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/math_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81082f28e1440f9d"}},{"code_sha256_prefix":"ed2f4689128f9885","entry":"extract_multi_choice_answer","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/parser.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed2f4689128f9885"}},{"code_sha256_prefix":"660713a8fa047736","entry":"get_generation_config","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"verl/utils/model.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"660713a8fa047736"}},{"code_sha256_prefix":"fe79c459a3e4cc64","entry":"get_huggingface_actor_config","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"verl/utils/model.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fe79c459a3e4cc64"}},{"code_sha256_prefix":"76f3c749a0e655f2","entry":"get_weight_loader","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"76f3c749a0e655f2"}},{"code_sha256_prefix":"fb37d3c84b18e368","entry":"is_digit","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/grader.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fb37d3c84b18e368"}},{"code_sha256_prefix":"1c1060f6e43d5dfa","entry":"is_number","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/math_utils.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/math_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1c1060f6e43d5dfa"}},{"code_sha256_prefix":"ee71a94d9dd1b8e9","entry":"load_hf_lm_and_tokenizer","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/model_utils.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee71a94d9dd1b8e9"}},{"code_sha256_prefix":"73925940858a62b3","entry":"my_parse_latex","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/math_utils.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/math_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73925940858a62b3"}},{"code_sha256_prefix":"7d4b7f8c3875b126","entry":"parse_digits","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"Qwen2.5-Eval/evaluation/grader.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/Qwen2.5-Eval/evaluation/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7d4b7f8c3875b126"}},{"code_sha256_prefix":"3b15e2ac7497c441","entry":"squeeze","repo":"ypwang61/one-shot-rlvr","repo_kind":"official","path":"verl/utils/model.py","file_url":"https://github.com/ypwang61/one-shot-rlvr/blob/HEAD/verl/utils/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3b15e2ac7497c441"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}