{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-12652","title":"Multi-Rollout On-Policy Distillation via Peer Successes and Failures","arxiv_id":"2605.12652","date":"2026-05-12","proceeding":null,"authors":["Weichen Yu","Xiaomin Li","Yizhou Zhao","Xiaoze Liu","Ruowang Zhang","Haixin Wang","Yinyi Luo","Chen Henry Wu","Gaurav Mittal","Matt Fredrikson","Yu Hu"],"abstract":"Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails. On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories, yet existing methods typically distill each rollout independently and ignore the other attempts sampled for the same prompt. We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals. MOPD conditions the teacher on both successful and failed peer rollouts: successes provide positive evidence for valid reasoning patterns, while failures provide structured negative evidence about plausible mistakes to avoid. We study two peer-context constructions: positive peer imitation and contrastive success-failure conditioning. Experiments on competitive programming, mathematical reasoning, scientific question answering, and tool-use benchmarks show that MOPD consistently improves over standard on-policy baselines. Further teacher-signal analysis shows that mixed success-failure contexts better align teacher scores with verifier rewards, indicating that the gains arise from more faithful, instance-adaptive supervision. These results indicate that effective on-policy distillation should exploit the student's multi-rollout trial-and-error behavior rather than treating rollouts as isolated samples.","url_abs":"https://arxiv.org/abs/2605.12652","url_pdf":"https://arxiv.org/pdf/2605.12652","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2605.12652","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.12652"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/viviable/mopd_code","reach":null}],"summary":{"ran":22,"unverified":7},"by_repo_kind":{"found_in_text":{"samples":29,"ran":22,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bd3548f55a4d47e5","entry":"average","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/compute_teacher_signal_metrics.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/compute_teacher_signal_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bd3548f55a4d47e5"}},{"code_sha256_prefix":"d7aa73af385d98f7","entry":"build_student_targets","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/build_target_responses.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/build_target_responses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d7aa73af385d98f7"}},{"code_sha256_prefix":"2d94e42b971ca757","entry":"check_correctness","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"verl/utils/reward_score/prime_code/utils.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/verl/utils/reward_score/prime_code/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d94e42b971ca757"}},{"code_sha256_prefix":"5ef8c6dd23fae513","entry":"ci_yerr","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/plot_teacher_metrics.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/plot_teacher_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5ef8c6dd23fae513"}},{"code_sha256_prefix":"288f33b5eca65cc7","entry":"condition_colors","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/plot_teacher_metrics.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/plot_teacher_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"288f33b5eca65cc7"}},{"code_sha256_prefix":"caa5d8692e5bdb76","entry":"fetch_history","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/draw_acc.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/draw_acc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"caa5d8692e5bdb76"}},{"code_sha256_prefix":"aebe25129f6e22f7","entry":"load_metrics","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/plot_teacher_metrics.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/plot_teacher_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aebe25129f6e22f7"}},{"code_sha256_prefix":"b97c9a97c01f9eea","entry":"make_client","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/generate_strong_success.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/generate_strong_success.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b97c9a97c01f9eea"}},{"code_sha256_prefix":"6339c418c950b9c8","entry":"only_int_check","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"verl/utils/reward_score/prime_code/testing_util.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/verl/utils/reward_score/prime_code/testing_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6339c418c950b9c8"}},{"code_sha256_prefix":"0898e734fc60f2e9","entry":"parse_run_path","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/draw_acc.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/draw_acc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0898e734fc60f2e9"}},{"code_sha256_prefix":"414365b67878b9e4","entry":"rankdata","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/compute_teacher_signal_metrics.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/compute_teacher_signal_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"414365b67878b9e4"}},{"code_sha256_prefix":"903f3f3a478c7f63","entry":"read_jsonl","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/score_teacher_contexts.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/score_teacher_contexts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"903f3f3a478c7f63"}},{"code_sha256_prefix":"2b6d83de04b5e4ad","entry":"read_jsonl","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/compute_teacher_signal_metrics.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/compute_teacher_signal_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2b6d83de04b5e4ad"}},{"code_sha256_prefix":"759afe3c4dcbc678","entry":"read_variant_config","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/build_context_variants.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/build_context_variants.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"759afe3c4dcbc678"}},{"code_sha256_prefix":"d9156c6e78fc7a12","entry":"resolve_api_config","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/generate_strong_success.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/generate_strong_success.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d9156c6e78fc7a12"}},{"code_sha256_prefix":"2d63c8da40e7f663","entry":"resolve_dtype","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/score_teacher_contexts.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/score_teacher_contexts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d63c8da40e7f663"}},{"code_sha256_prefix":"53fb0fcdb63f7f14","entry":"resolve_run","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/fetch_wandb_metric_compare.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/fetch_wandb_metric_compare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"53fb0fcdb63f7f14"}},{"code_sha256_prefix":"6f3c534aa4ba4ee2","entry":"sanitize","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/fetch_wandb_metric_compare.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/fetch_wandb_metric_compare.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f3c534aa4ba4ee2"}},{"code_sha256_prefix":"f5674775fedebe04","entry":"sanitize_metric_name","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/draw_acc.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/draw_acc.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f5674775fedebe04"}},{"code_sha256_prefix":"2d00b81e871f020d","entry":"stable_target_id","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/build_target_responses.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/build_target_responses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d00b81e871f020d"}},{"code_sha256_prefix":"3d4ee5bf4a4b62a8","entry":"stable_variant_id","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/build_context_variants.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/build_context_variants.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3d4ee5bf4a4b62a8"}},{"code_sha256_prefix":"22849627aa42fa47","entry":"truncatefn","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"verl/utils/reward_score/prime_code/testing_util.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/verl/utils/reward_score/prime_code/testing_util.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22849627aa42fa47"}},{"code_sha256_prefix":"1a52c3aa3027b3fe","entry":"load_model_and_tokenizer","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/score_teacher_contexts.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/score_teacher_contexts.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1a52c3aa3027b3fe"}},{"code_sha256_prefix":"984827baf66eba8d","entry":"load_spec","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/plot_heatmap_grid.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/plot_heatmap_grid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"984827baf66eba8d"}},{"code_sha256_prefix":"b29f33cf4c2111e6","entry":"resolve_run_path","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"analysis/fetch_wandb_metric_compare.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/analysis/fetch_wandb_metric_compare.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b29f33cf4c2111e6"}},{"code_sha256_prefix":"45691de9cab1a113","entry":"run_test_code","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"verl/utils/reward_score/feedback/code.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/verl/utils/reward_score/feedback/code.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"45691de9cab1a113"}},{"code_sha256_prefix":"4a71e4f42656119d","entry":"run_test_func","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"verl/utils/reward_score/feedback/code.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/verl/utils/reward_score/feedback/code.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4a71e4f42656119d"}},{"code_sha256_prefix":"f1b61b9749cac892","entry":"run_test_std","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"verl/utils/reward_score/feedback/code.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/verl/utils/reward_score/feedback/code.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f1b61b9749cac892"}},{"code_sha256_prefix":"0b462a89ce04228a","entry":"string_int_check","repo":"viviable/mopd_code","repo_kind":"found_in_text","path":"verl/utils/reward_score/prime_code/testing_util.py","file_url":"https://github.com/viviable/mopd_code/blob/HEAD/verl/utils/reward_score/prime_code/testing_util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0b462a89ce04228a"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}