{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/concise-reasoning-via-reinforcement-learning","title":"Concise Reasoning via Reinforcement Learning","arxiv_id":"2504.05185","date":"2025-04-07","proceeding":null,"authors":["Mehdi Fatemi","Banafsheh Rafiee","Mingjie Tang","Kartik Talamadupula"],"abstract":"Despite significant advancements in large language models (LLMs), a major drawback of reasoning models is their enormous token usage, which increases computational cost, resource requirements, and response time. In this work, we revisit the core principles of reinforcement learning (RL) and, through mathematical analysis, demonstrate that the tendency to generate lengthy responses arises inherently from RL-based optimization during training. This finding questions the prevailing assumption that longer responses inherently improve reasoning accuracy. Instead, we uncover a natural correlation between conciseness and accuracy that has been largely overlooked. We show that introducing a secondary phase of RL training, using a very small set of problems, can significantly reduce chains of thought while maintaining or even enhancing accuracy. Additionally, we demonstrate that, while GRPO shares some interesting properties of PPO, it suffers from collapse modes, which limit its reliability for concise reasoning. Finally, we validate our conclusions through extensive experimental results.","url_abs":"https://arxiv.org/abs/2504.05185v2","url_pdf":"https://arxiv.org/pdf/2504.05185v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"concise-reasoning-via-reinforcement-learning","repo_url":"https://github.com/ai-wand/concise-reasoning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"entropy-regularization","method_name":"Entropy Regularization"},{"method_slug":"ppo","method_name":"PPO"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.05185","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.05185"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ai-wand/concise-reasoning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"ran_violates":1,"ran_draft_wrong":1,"unverified":11},"by_repo_kind":{"official":{"samples":13,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":1,"samples":[{"code_sha256_prefix":"1525d8bc0fc8d5a0","entry":"generate_completions","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/model_utils.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/model_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1525d8bc0fc8d5a0"}},{"code_sha256_prefix":"d338a8cb44aeb6b4","entry":"is_multi_choice","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/math_eval.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/math_eval.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d338a8cb44aeb6b4"}},{"code_sha256_prefix":"18ed61dcfadbbdf0","entry":"reward_fn","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"18ed61dcfadbbdf0"}},{"code_sha256_prefix":"809c0178a7ae90ff","entry":"choice_answer_clean","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/grader.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"809c0178a7ae90ff"}},{"code_sha256_prefix":"81082f28e1440f9d","entry":"compare_numerical_ans","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/math_utils.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/math_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"81082f28e1440f9d"}},{"code_sha256_prefix":"ed2f4689128f9885","entry":"extract_multi_choice_answer","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/parser.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/parser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed2f4689128f9885"}},{"code_sha256_prefix":"fb37d3c84b18e368","entry":"is_digit","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/grader.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fb37d3c84b18e368"}},{"code_sha256_prefix":"581f3949cfdcffa2","entry":"is_execution_success","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/trajectory.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/trajectory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"581f3949cfdcffa2"}},{"code_sha256_prefix":"1c1060f6e43d5dfa","entry":"is_number","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/math_utils.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/math_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c1060f6e43d5dfa"}},{"code_sha256_prefix":"ee71a94d9dd1b8e9","entry":"load_hf_lm_and_tokenizer","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/model_utils.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/model_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee71a94d9dd1b8e9"}},{"code_sha256_prefix":"73925940858a62b3","entry":"my_parse_latex","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/math_utils.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/math_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73925940858a62b3"}},{"code_sha256_prefix":"7d4b7f8c3875b126","entry":"parse_digits","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/grader.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/grader.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7d4b7f8c3875b126"}},{"code_sha256_prefix":"7e48538d19cc0d8f","entry":"text_to_trajectory","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/trajectory.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/trajectory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e48538d19cc0d8f"}},{"code_sha256_prefix":"0d7ae684ec6c7537","entry":"trajectory_to_text","repo":"ai-wand/concise-reasoning","repo_kind":"official","path":"eval/trajectory.py","file_url":"https://github.com/ai-wand/concise-reasoning/blob/HEAD/eval/trajectory.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d7ae684ec6c7537"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}