{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-19236","title":"STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability","arxiv_id":"2606.19236","date":"2026-06-17","proceeding":null,"authors":["Haipeng Luo","Qingfeng Sun","Songli Wu","Can Xu","Wenfeng Deng","Han Hu","Yansong Tang"],"abstract":"Reinforcement Learning with Verifiable Rewards algorithms like GRPO have emerged as the dominant post-training paradigm for complex reasoning in LLMs, yet commonly suffer from policy entropy collapse during training. We conduct a first-order gradient analysis of token-level entropy dynamics under GRPO and identify a token-level credit assignment mismatch: the per-token entropy variation decomposes into the product of the trajectory-level advantage and an entropy sensitivity function over the next-token distribution, yielding an advantage-surprisal four-quadrant structure and a near-criticality property. Motivated by it, we propose STARE (Surprisal-guided Token-level Advantage Reweighting for policy Entropy stability), which identifies entropy-critical token subsets via batch-internal surprisal quantiles, selectively reweights their effective advantages, and incorporates a target-entropy closed-loop gate for stable entropy regulation. Across model scales from 1.5B to 32B and three task families (Short CoT, Long CoT, and Multi-Turn Tool Use), STARE sustains stable RL training over thousands of steps while maintaining policy entropy within the target band. On AIME24 and AIME25, STARE outperforms DAPO and other competitive baselines by 4%-8% in average accuracy, with reflection tokens and response length growing in tandem, indicating sustained exploration-exploitation balance that further unlocks RL training potential.Code is available at https://github.com/hp-luo/STARE.","url_abs":"https://arxiv.org/abs/2606.19236","url_pdf":"https://arxiv.org/pdf/2606.19236","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.19236","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.19236"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/hp-luo/STARE","reach":null}],"summary":{"ran":10,"ran_fixture":1},"by_repo_kind":{"found_in_text":{"samples":11,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f220595adfc5e83c","entry":"compute_ce_dpo_loss_rm","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"recipe/prime/prime_core_algos.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f220595adfc5e83c"}},{"code_sha256_prefix":"4d91e4383716e42c","entry":"compute_detach_dpo_loss_rm","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"recipe/prime/prime_core_algos.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d91e4383716e42c"}},{"code_sha256_prefix":"f19825983a523003","entry":"compute_online_dpo_loss","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"recipe/spin/core_algos.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f19825983a523003"}},{"code_sha256_prefix":"41b6e81dc3ecba79","entry":"compute_onlinedpo_pref","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"recipe/spin/core_algos.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41b6e81dc3ecba79"}},{"code_sha256_prefix":"05a86ccb0b3876dd","entry":"compute_score_data_source","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"recipe/open_math_reasoning/compute_score.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/recipe/open_math_reasoning/compute_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"05a86ccb0b3876dd"}},{"code_sha256_prefix":"d138cde65a9c4401","entry":"extract_solution","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"verl/utils/reward_score/math_stare.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/verl/utils/reward_score/math_stare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d138cde65a9c4401"}},{"code_sha256_prefix":"757d109d90121142","entry":"get_kl_controller","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"recipe/spin/core_algos.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"757d109d90121142"}},{"code_sha256_prefix":"11c8e99168eeef2e","entry":"normalize","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"recipe/gkd/megatron_kl_loss.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/recipe/gkd/megatron_kl_loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11c8e99168eeef2e"}},{"code_sha256_prefix":"2aa611c91f5e4478","entry":"pred_ans_extract","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"verl/utils/reward_score/math_stare.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/verl/utils/reward_score/math_stare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2aa611c91f5e4478"}},{"code_sha256_prefix":"1305c915f184b3a4","entry":"pred_ans_extract_train","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"verl/utils/reward_score/math_stare.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/verl/utils/reward_score/math_stare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1305c915f184b3a4"}},{"code_sha256_prefix":"8a84f8cc921357e2","entry":"reward_func","repo":"hp-luo/STARE","repo_kind":"found_in_text","path":"recipe/r1/reward_score.py","file_url":"https://github.com/hp-luo/STARE/blob/HEAD/recipe/r1/reward_score.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a84f8cc921357e2"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}