{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-17928","title":"HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignment","arxiv_id":"2604.17928","date":"2026-04-20","proceeding":null,"authors":["Zhanyu Liu","Qingguo Hu","Ante Wang","Chenqing Liu","Zhishang Xiang","Hui Li","Delai Qiu","Jinsong Su"],"abstract":"Reinforcement Learning with Verifiable Reward (RLVR) has proven effective for training reasoning-oriented large language models, but existing methods largely assume high-resource settings with abundant training data. In low-resource scenarios, RLVR is prone to more severe entropy collapse, which substantially limits exploration and degrades reasoning performance. To address this issue, we propose Hybrid-domain Entropy dynamics ALignment (HEAL), a framework tailored for few-shot RLVR. HEAL first selectively incorporates high-value general-domain data to promote more diverse exploration. Then, we introduce Entropy Dynamics Alignment (EDA), a reward mechanism that aligns trajectory-level entropy dynamics between the target and general domains, capturing both entropy magnitude and fine-grained variation. Through this alignment, EDA not only further mitigates entropy collapse but also encourages the policy to acquire more diverse exploration behaviors from the general domain. Experiments across multiple domains show that HEAL consistently improves few-shot RLVR performance. Notably, using only 32 target-domain samples, HEAL matches or even surpasses full-shot RLVR trained with 1K target-domain samples.","url_abs":"https://arxiv.org/abs/2604.17928","url_pdf":"https://arxiv.org/pdf/2604.17928","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.17928","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.17928"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/XMUDeepLIT/HEAL","reach":null}],"summary":{"ran":8,"unverified":5},"by_repo_kind":{"found_in_text":{"samples":13,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f220595adfc5e83c","entry":"compute_ce_dpo_loss_rm","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"recipe/prime/prime_core_algos.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f220595adfc5e83c"}},{"code_sha256_prefix":"4d91e4383716e42c","entry":"compute_detach_dpo_loss_rm","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"recipe/prime/prime_core_algos.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d91e4383716e42c"}},{"code_sha256_prefix":"f19825983a523003","entry":"compute_online_dpo_loss","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"recipe/spin/core_algos.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f19825983a523003"}},{"code_sha256_prefix":"41b6e81dc3ecba79","entry":"compute_onlinedpo_pref","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"recipe/spin/core_algos.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41b6e81dc3ecba79"}},{"code_sha256_prefix":"9c496dae615ba2a8","entry":"generate_config_from_args","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"verl/model_merger/base_model_merger.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/verl/model_merger/base_model_merger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9c496dae615ba2a8"}},{"code_sha256_prefix":"757d109d90121142","entry":"get_kl_controller","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"recipe/spin/core_algos.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"757d109d90121142"}},{"code_sha256_prefix":"c30b2a96103abd2c","entry":"reward_func","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"recipe/r1/reward_score.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/recipe/r1/reward_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c30b2a96103abd2c"}},{"code_sha256_prefix":"83b1686d7ac03c21","entry":"unpad_dataproto","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"verl/protocol.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/verl/protocol.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"83b1686d7ac03c21"}},{"code_sha256_prefix":"acf240079d434746","entry":"get_weight_loader","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"acf240079d434746"}},{"code_sha256_prefix":"cd6b1e0d6afb2762","entry":"get_weight_saver","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"verl/models/weight_loader_registry.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/verl/models/weight_loader_registry.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd6b1e0d6afb2762"}},{"code_sha256_prefix":"461e5db81919f588","entry":"parallel_compute_score_async","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"verl/workers/reward_manager/prime_heal.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/verl/workers/reward_manager/prime_heal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"461e5db81919f588"}},{"code_sha256_prefix":"be0eac986f33339f","entry":"run_reward_scoring","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"verl/workers/reward_manager/prime_heal.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/verl/workers/reward_manager/prime_heal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be0eac986f33339f"}},{"code_sha256_prefix":"2e7925fe4691d63f","entry":"single_compute_score","repo":"XMUDeepLIT/HEAL","repo_kind":"found_in_text","path":"verl/workers/reward_manager/prime_heal.py","file_url":"https://github.com/XMUDeepLIT/HEAL/blob/HEAD/verl/workers/reward_manager/prime_heal.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2e7925fe4691d63f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}