{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-07711","title":"SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation","arxiv_id":"2605.07711","date":"2026-05-08","proceeding":null,"authors":["Jie Sun","Mao Zheng","Mingyang Song","Qiyong Zhong","Yilin Cheng","Bichuan Feng","Pengfei Liu","Junfeng Fang","Xiang Wang"],"abstract":"On-policy distillation (OPD) is a standard tool for transferring teacher behavior to a smaller student, but it implicitly assumes that teacher and student predictions are comparable token by token, an assumption that fails whenever the two models tokenize the same text differently. Under heterogeneous tokenizers, exact shared-token matching silently discards a large fraction of the teacher signal at precisely the positions where vocabularies disagree. We propose \\textbf{\\underline{Sim}ple \\underline{C}ross-\\underline{T}okenizer OPD (SimCT)}, which restores this signal by enlarging the supervision space: alongside shared tokens, SimCT compares teacher and student over short multi-token continuations that both tokenizers can realize, leaving the OPD loss form itself unchanged. We show that these units are the finest jointly tokenizable supervision interface, and that coarser alternatives remove teacher-student distinctions that are useful for on-policy learning. Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT shows consistent gains over shared-vocabulary OPD and representative cross-tokenizer baselines, with ablations confirming that the improvements come from recovering supervision discarded by exact shared-token matching. Code is available at \\href{https://github.com/sunjie279/SimCT-}{https://github.com/sunjie279/SimCT-}.","url_abs":"https://arxiv.org/abs/2605.07711","url_pdf":"https://arxiv.org/pdf/2605.07711","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2605.07711","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.07711"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/sunjie279/SimCT-","reach":null}],"summary":{"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":4,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"31ccedc9e2356412","entry":"register_algorithm","repo":"sunjie279/SimCT-","repo_kind":"found_in_text","path":"kdflow/algorithms/span_ctkd.py","file_url":"https://github.com/sunjie279/SimCT-/blob/HEAD/kdflow/algorithms/span_ctkd.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31ccedc9e2356412"}},{"code_sha256_prefix":"dbc4b17b604772c4","entry":"SpanCrossTokenizerKD","repo":"sunjie279/SimCT-","repo_kind":"found_in_text","path":"kdflow/algorithms/span_ctkd.py","file_url":"https://github.com/sunjie279/SimCT-/blob/HEAD/kdflow/algorithms/span_ctkd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbc4b17b604772c4"}},{"code_sha256_prefix":"699306a58a881dbf","entry":"build_loss_fn","repo":"sunjie279/SimCT-","repo_kind":"found_in_text","path":"kdflow/algorithms/span_ctkd.py","file_url":"https://github.com/sunjie279/SimCT-/blob/HEAD/kdflow/algorithms/span_ctkd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"699306a58a881dbf"}},{"code_sha256_prefix":"873c7b566efd502a","entry":"compute_cross_entropy","repo":"sunjie279/SimCT-","repo_kind":"found_in_text","path":"kdflow/algorithms/span_ctkd.py","file_url":"https://github.com/sunjie279/SimCT-/blob/HEAD/kdflow/algorithms/span_ctkd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"873c7b566efd502a"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}