{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cross-tokenizer-distillation-via-approximate","title":"Cross-Tokenizer Distillation via Approximate Likelihood Matching","arxiv_id":"2503.20083","date":"2025-03-25","proceeding":null,"authors":["Benjamin Minixhofer","Ivan Vulić","Edoardo Maria Ponti"],"abstract":"Distillation has shown remarkable success in transferring knowledge from a Large Language Model (LLM) teacher to a student LLM. However, current distillation methods predominantly require the same tokenizer between the teacher and the student, restricting their applicability to only a small subset of teacher-student pairs. In this work, we develop a cross-tokenizer distillation method to solve this crucial deficiency. Our method is the first to enable cross-tokenizer distillation without a next-token prediction loss as the main objective, instead purely maximizing the student predictions' similarity to the teacher's predictions (known as pure distillation), while also being robust to large mismatches between the teacher and the student tokenizer function and vocabulary. Empirically, our method enables substantially improved performance as tested on two use cases. First, we show that viewing tokenizer transfer as self-distillation enables unprecedently effective transfer across tokenizers. We transfer (subword-level) Llama and Gemma models to byte-level tokenization more effectively than prior methods transfer to a similar subword tokenizer under a comparable training budget. Transferring different base models to the same tokenizer also enables ensembling them (e.g., via averaging their predicted probabilities) which boosts performance. Second, we use our cross-tokenizer distillation method to distil a large maths-specialized LLM into a smaller model, achieving competitive maths problem-solving performance. Overall, our results make substantial strides toward better adaptability and enhanced interaction between different LLMs.","url_abs":"https://arxiv.org/abs/2503.20083v2","url_pdf":"https://arxiv.org/pdf/2503.20083v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cross-tokenizer-distillation-via-approximate","repo_url":"https://github.com/bminixhofer/alm","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.20083","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.20083"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/bminixhofer/tokenkit","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bminixhofer/alm","reach":null}],"summary":{"ran_honours":1,"ran_violates":1,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1},"found_in_text":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d6ce06c44dad6c4e","entry":"get_last_index_per_column","repo":"bminixhofer/alm","repo_kind":"official","path":"tokenkit/training/losses.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/training/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d6ce06c44dad6c4e"}},{"code_sha256_prefix":"291bf36d858a5fae","entry":"log1mexp","repo":"bminixhofer/alm","repo_kind":"official","path":"tokenkit/training/losses.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/training/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"291bf36d858a5fae"}},{"code_sha256_prefix":"62b54542902a6a11","entry":"pad_embeddings_with_random","repo":"bminixhofer/tokenkit","repo_kind":"found_in_text","path":"scripts/cross_tokenizer_distill.py","file_url":"https://github.com/bminixhofer/tokenkit/blob/HEAD/scripts/cross_tokenizer_distill.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62b54542902a6a11"}},{"code_sha256_prefix":"7828f95cb9e3bc8b","entry":"compute_alm_loss","repo":"bminixhofer/alm","repo_kind":"official","path":"tokenkit/training/losses.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/training/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7828f95cb9e3bc8b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}