{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-07079","title":"Entropy-Aware On-Policy Distillation of Language Models","arxiv_id":"2603.07079","date":"2026-03-07","proceeding":"ICML","authors":["Woogyeol Jin","Taywon Min","Yongjin Yang","Dennis Wei","Yi Zhou","Swanand Ravindra Kadhe","Nathalie Baracaldo","Kimin Lee"],"abstract":"On-policy distillation is a promising approach for transferring knowledge between language models, where a student learns from dense token-level signals along its own trajectories. This framework typically uses reverse KL divergence, encouraging the student to match the teacher's high-confidence predictions. However, we show that the mode-seeking property of reverse KL reduces generation diversity and yields unstable learning signals when the teacher distribution has high entropy. To address this, we introduce Entropy-Aware On-Policy Distillation. Our key idea is augmenting the standard reverse KL objective with forward KL when teacher entropy is high, capturing the full range of plausible outputs while retaining precise imitation elsewhere. It balances mode-seeking precision with mode-covering robustness without sacrificing on-policy training efficiency. Experiments show that our method maintains generation diversity (sustained token-level entropy) and improves student-teacher alignment (lower forward KL on high-entropy tokens). Across six math reasoning benchmarks, this yields Pass@8 accuracy gains of +1.37 for Qwen3-0.6B-Base, +2.39 for Qwen3-1.7B-Base, and +5.05 for Qwen3-4B-Base compared to baseline on-policy distillation methods. These results demonstrate that accounting for teacher uncertainty is essential for maintaining diversity and achieving effective knowledge transfer.","url_abs":"https://arxiv.org/abs/2603.07079","url_pdf":"https://arxiv.org/pdf/2603.07079","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.07079","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.07079"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/WLS04/EOPD","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"ran_fixture":1},"by_repo_kind":{"found_in_text":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f220595adfc5e83c","entry":"compute_ce_dpo_loss_rm","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"recipe/prime/prime_core_algos.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f220595adfc5e83c"}},{"code_sha256_prefix":"4d91e4383716e42c","entry":"compute_detach_dpo_loss_rm","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"recipe/prime/prime_core_algos.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/recipe/prime/prime_core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4d91e4383716e42c"}},{"code_sha256_prefix":"f19825983a523003","entry":"compute_online_dpo_loss","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"recipe/spin/core_algos.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f19825983a523003"}},{"code_sha256_prefix":"41b6e81dc3ecba79","entry":"compute_onlinedpo_pref","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"recipe/spin/core_algos.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"41b6e81dc3ecba79"}},{"code_sha256_prefix":"05a86ccb0b3876dd","entry":"compute_score_data_source","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"recipe/open_math_reasoning/compute_score.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/recipe/open_math_reasoning/compute_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"05a86ccb0b3876dd"}},{"code_sha256_prefix":"8896f4e2bafe8abc","entry":"generate_config_from_args","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"verl/model_merger/base_model_merger.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/verl/model_merger/base_model_merger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8896f4e2bafe8abc"}},{"code_sha256_prefix":"94209884dcb2308e","entry":"get_dynamic_pipeline_shards","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"verl/model_merger/megatron_model_merger.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/verl/model_merger/megatron_model_merger.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94209884dcb2308e"}},{"code_sha256_prefix":"757d109d90121142","entry":"get_kl_controller","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"recipe/spin/core_algos.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/recipe/spin/core_algos.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"757d109d90121142"}},{"code_sha256_prefix":"11c8e99168eeef2e","entry":"normalize","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"recipe/gkd/megatron_kl_loss.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/recipe/gkd/megatron_kl_loss.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"11c8e99168eeef2e"}},{"code_sha256_prefix":"8a84f8cc921357e2","entry":"reward_func","repo":"WLS04/EOPD","repo_kind":"found_in_text","path":"recipe/r1/reward_score.py","file_url":"https://github.com/WLS04/EOPD/blob/HEAD/recipe/r1/reward_score.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"DEP_MISSING","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8a84f8cc921357e2"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}