{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-28123","title":"Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL","arxiv_id":"2604.28123","date":"2026-04-30","proceeding":null,"authors":["Sudong Wang","Weiquan Huang","Xiaomin Yu","Zuhao Yang","Hehai Lin","Keming Wu","Chaojun Xiao","Chen Chen","Wenxuan Wang","Beier Zhu","Yunjian Zhang","Chengwei Qin"],"abstract":"The standard post-training recipe for large multimodal models (LMMs) applies supervised fine-tuning (SFT) on curated demonstrations followed by reinforcement learning with verifiable rewards (RLVR). However, SFT introduces distributional drift that neither preserves the model's original capabilities nor faithfully matches the supervision distribution. This problem is further amplified in multimodal reasoning, where perception errors and reasoning failures follow distinct drift patterns that compound during subsequent RL. We introduce PRISM, a three-stage pipeline that mitigates this drift by inserting an explicit distribution-alignment stage between SFT and RLVR. Building on the principle of on-policy distillation (OPD), PRISM casts alignment as a black-box, response-level adversarial game between the policy and a Mixture-of-Experts (MoE) discriminator with dedicated perception and reasoning experts, providing disentangled corrective signals that steer the policy toward the supervision distribution without requiring access to teacher logits. While 1.26M public demonstrations suffice for broad SFT initialization, distribution alignment demands higher-fidelity supervision; we therefore curate 113K additional demonstrations from Gemini 3 Flash, featuring dense visual grounding and step-by-step reasoning on the hardest unsolved problems. Experiments on Qwen3-VL show that PRISM consistently improves downstream RLVR performance across multiple RL algorithms (GRPO, DAPO, GSPO) and diverse multimodal benchmarks, improving average accuracy by +4.4 and +6.0 points over the SFT-to-RLVR baseline on 4B and 8B, respectively. Our code, data, and model checkpoints are publicly available at https://github.com/XIAO4579/PRISM.","url_abs":"https://arxiv.org/abs/2604.28123","url_pdf":"https://arxiv.org/pdf/2604.28123","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.28123","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.28123"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/XIAO4579/PRISM","reach":null}],"summary":{"ran":7,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":11,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d36fc4b002a2590e","entry":"collate_fn","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/dense_model/train_dense_vl_warmup.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/dense_model/train_dense_vl_warmup.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d36fc4b002a2590e"}},{"code_sha256_prefix":"9c7b92287d75fe98","entry":"convert_safetensors_to_pt","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/convert_value_head.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/convert_value_head.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c7b92287d75fe98"}},{"code_sha256_prefix":"ee32a88292584e6b","entry":"extract_value_head","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/extract_value_head.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/extract_value_head.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ee32a88292584e6b"}},{"code_sha256_prefix":"c11b99400c6e3f56","entry":"get_last_token_scores","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/dense_model/value_head.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/dense_model/value_head.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c11b99400c6e3f56"}},{"code_sha256_prefix":"c6e48cbf2dfb014e","entry":"load_image","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/dense_model/train_dense_vl_warmup.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/dense_model/train_dense_vl_warmup.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c6e48cbf2dfb014e"}},{"code_sha256_prefix":"8927c27669d0e910","entry":"process_sample","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/inference_2_vl_moe.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/inference_2_vl_moe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8927c27669d0e910"}},{"code_sha256_prefix":"35a9e9f1edf6e6c1","entry":"process_sample","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/inference_vl_moe.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/inference_vl_moe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"35a9e9f1edf6e6c1"}},{"code_sha256_prefix":"c80cfd02a636fcf2","entry":"test_image_forward","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/create_2_vl_moe.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/create_2_vl_moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c80cfd02a636fcf2"}},{"code_sha256_prefix":"76fd4d8019fa116b","entry":"test_text_only_forward","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/create_2_vl_moe.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/create_2_vl_moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"76fd4d8019fa116b"}},{"code_sha256_prefix":"ea98384919e02f3a","entry":"upcycle_qwen3_to_moe","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/create_moe.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/create_moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ea98384919e02f3a"}},{"code_sha256_prefix":"016994fd5d99afdc","entry":"upcycle_qwen3_vl_to_moe","repo":"XIAO4579/PRISM","repo_kind":"found_in_text","path":"moe/create_vl_moe.py","file_url":"https://github.com/XIAO4579/PRISM/blob/HEAD/moe/create_vl_moe.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"016994fd5d99afdc"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}