{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/toolrl-reward-is-all-tool-learning-needs","title":"ToolRL: Reward is All Tool Learning Needs","arxiv_id":"2504.13958","date":"2025-04-16","proceeding":null,"authors":["Cheng Qian","Emre Can Acikgoz","Qi He","Hongru Wang","Xiusi Chen","Dilek Hakkani-Tür","Gokhan Tur","Heng Ji"],"abstract":"Current Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to unfamiliar or complex tool use scenarios. Recent advancements in reinforcement learning (RL), particularly with R1-like models, have demonstrated promising reasoning and generalization abilities. Yet, reward design for tool use presents unique challenges: multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning. In this work, we present the first comprehensive study on reward design for tool selection and application tasks within the RL paradigm. We systematically explore a wide range of reward strategies, analyzing their types, scales, granularity, and temporal dynamics. Building on these insights, we propose a principled reward design tailored for tool use tasks and apply it to train LLMs using Group Relative Policy Optimization (GRPO). Empirical evaluations across diverse benchmarks demonstrate that our approach yields robust, scalable, and stable training, achieving a 17% improvement over base models and a 15% gain over SFT models. These results highlight the critical role of thoughtful reward design in enhancing the tool use capabilities and generalization performance of LLMs. All the codes are released to facilitate future research.","url_abs":"https://arxiv.org/abs/2504.13958v1","url_pdf":"https://arxiv.org/pdf/2504.13958v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"toolrl-reward-is-all-tool-learning-needs","repo_url":"https://github.com/qiancheng0/toolrl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.13958","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.13958"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/qiancheng0/toolrl","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":2,"ran_fixture":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e8c21fee70ac35b0","entry":"customize_format_reward_func","repo":"qiancheng0/toolrl","repo_kind":"official","path":"verl/utils/reward_score/rlla.py","file_url":"https://github.com/qiancheng0/toolrl/blob/HEAD/verl/utils/reward_score/rlla.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8c21fee70ac35b0"}},{"code_sha256_prefix":"915551c6ca0ec81f","entry":"customize_length_reward_func","repo":"qiancheng0/toolrl","repo_kind":"official","path":"verl/utils/reward_score/rlla.py","file_url":"https://github.com/qiancheng0/toolrl/blob/HEAD/verl/utils/reward_score/rlla.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"915551c6ca0ec81f"}},{"code_sha256_prefix":"6281513745ecdf4e","entry":"match_score","repo":"qiancheng0/toolrl","repo_kind":"official","path":"verl/utils/reward_score/rlla.py","file_url":"https://github.com/qiancheng0/toolrl/blob/HEAD/verl/utils/reward_score/rlla.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6281513745ecdf4e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}