{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-04597","title":"Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning","arxiv_id":"2603.04597","date":"2026-03-04","proceeding":null,"authors":["Lei Huang","Xiang Cheng","Chenxiao Zhao","Guobin Shen","Junjie Yang","Xiaocheng Feng","Yuxuan Gu","Xing Yu","Bing Qin"],"abstract":"Large language models (LLMs) typically receive diverse natural language (NL) feedback through interaction with the environment. However, current reinforcement learning (RL) algorithms rely solely on scalar rewards, leaving the rich information in NL feedback underutilized and leading to inefficient exploration. In this work, we propose GOLF, an RL framework that explicitly exploits group-level language feedback to guide targeted exploration through actionable refinements. GOLF aggregates two complementary feedback sources: (i) external critiques that pinpoint errors or propose targeted fixes, and (ii) intra-group attempts that supply alternative partial ideas and diverse failure patterns. These group-level feedbacks are aggregated to produce high-quality refinements, which are adaptively injected into training as off-policy scaffolds to provide targeted guidance in sparse-reward regions. Meanwhile, GOLF jointly optimizes generation and refinement within a unified RL loop, creating a virtuous cycle that continuously improves both capabilities. Experiments on both verifiable and non-verifiable benchmarks show that GOLF achieves superior performance and exploration efficiency, achieving 2.2$\\times$ improvements in sample efficiency compared to RL methods trained solely on scalar rewards. Code is available at https://github.com/LuckyyySTA/GOLF.","url_abs":"https://arxiv.org/abs/2603.04597","url_pdf":"https://arxiv.org/pdf/2603.04597","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.04597","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.04597"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/LuckyyySTA/GOLF","reach":null}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f220595adfc5e83c","entry":"compute_ce_dpo_loss_rm","repo":"LuckyyySTA/GOLF","repo_kind":"found_in_text","path":"golf/verl/recipe/prime/prime_core_algos.py","file_url":"https://github.com/LuckyyySTA/GOLF/blob/HEAD/golf/verl/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f220595adfc5e83c"}},{"code_sha256_prefix":"4d91e4383716e42c","entry":"compute_detach_dpo_loss_rm","repo":"LuckyyySTA/GOLF","repo_kind":"found_in_text","path":"golf/verl/recipe/prime/prime_core_algos.py","file_url":"https://github.com/LuckyyySTA/GOLF/blob/HEAD/golf/verl/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d91e4383716e42c"}},{"code_sha256_prefix":"f19825983a523003","entry":"compute_online_dpo_loss","repo":"LuckyyySTA/GOLF","repo_kind":"found_in_text","path":"golf/verl/recipe/spin/core_algos.py","file_url":"https://github.com/LuckyyySTA/GOLF/blob/HEAD/golf/verl/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f19825983a523003"}},{"code_sha256_prefix":"41b6e81dc3ecba79","entry":"compute_onlinedpo_pref","repo":"LuckyyySTA/GOLF","repo_kind":"found_in_text","path":"golf/verl/recipe/spin/core_algos.py","file_url":"https://github.com/LuckyyySTA/GOLF/blob/HEAD/golf/verl/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41b6e81dc3ecba79"}},{"code_sha256_prefix":"757d109d90121142","entry":"get_kl_controller","repo":"LuckyyySTA/GOLF","repo_kind":"found_in_text","path":"golf/verl/recipe/spin/core_algos.py","file_url":"https://github.com/LuckyyySTA/GOLF/blob/HEAD/golf/verl/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"757d109d90121142"}},{"code_sha256_prefix":"c30b2a96103abd2c","entry":"reward_func","repo":"LuckyyySTA/GOLF","repo_kind":"found_in_text","path":"golf/verl/recipe/r1/reward_score.py","file_url":"https://github.com/LuckyyySTA/GOLF/blob/HEAD/golf/verl/recipe/r1/reward_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c30b2a96103abd2c"}},{"code_sha256_prefix":"b7250010ea994bcd","entry":"accuracy_reward","repo":"LuckyyySTA/GOLF","repo_kind":"found_in_text","path":"golf/verl/verl/adaptive_mix_src/reward_with_format.py","file_url":"https://github.com/LuckyyySTA/GOLF/blob/HEAD/golf/verl/verl/adaptive_mix_src/reward_with_format.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b7250010ea994bcd"}},{"code_sha256_prefix":"e90d214897d14000","entry":"math_verify_reward_fn","repo":"LuckyyySTA/GOLF","repo_kind":"found_in_text","path":"golf/verl/verl/adaptive_mix_src/reward_with_format.py","file_url":"https://github.com/LuckyyySTA/GOLF/blob/HEAD/golf/verl/verl/adaptive_mix_src/reward_with_format.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e90d214897d14000"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}