{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zero-shot-tokenizer-transfer","title":"Zero-Shot Tokenizer Transfer","arxiv_id":"2405.07883","date":"2024-05-13","proceeding":null,"authors":["Benjamin Minixhofer","Edoardo Maria Ponti","Ivan Vulić"],"abstract":"Language models (LMs) are bound to their tokenizer, which maps raw text to a sequence of vocabulary items (tokens). This restricts their flexibility: for example, LMs trained primarily on English may still perform well in other natural and programming languages, but have vastly decreased efficiency due to their English-centric tokenizer. To mitigate this, we should be able to swap the original LM tokenizer with an arbitrary one, on the fly, without degrading performance. Hence, in this work we define a new problem: Zero-Shot Tokenizer Transfer (ZeTT). The challenge at the core of ZeTT is finding embeddings for the tokens in the vocabulary of the new tokenizer. Since prior heuristics for initializing embeddings often perform at chance level in a ZeTT setting, we propose a new solution: we train a hypernetwork taking a tokenizer as input and predicting the corresponding embeddings. We empirically demonstrate that the hypernetwork generalizes to new tokenizers both with encoder (e.g., XLM-R) and decoder LLMs (e.g., Mistral-7B). Our method comes close to the original models' performance in cross-lingual and coding tasks while markedly reducing the length of the tokenized sequence. We also find that the remaining gap can be quickly closed by continued training on less than 1B tokens. Finally, we show that a ZeTT hypernetwork trained for a base (L)LM can also be applied to fine-tuned variants without extra training. Overall, our results make substantial strides toward detaching LMs from their tokenizer.","url_abs":"https://arxiv.org/abs/2405.07883v1","url_pdf":"https://arxiv.org/pdf/2405.07883v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zero-shot-tokenizer-transfer","repo_url":"https://github.com/bminixhofer/zett","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}},{"paper_slug":"zero-shot-tokenizer-transfer","repo_url":"https://github.com/bminixhofer/alm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":null}],"tasks":[{"task_slug":"xlm-r","task_name":"XLM-R"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"hypernetwork","method_name":"HyperNetwork"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.07883","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.07883"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/InflectionAI/Inflection-Benchmarks","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bminixhofer/alm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bminixhofer/zett","reach":{"status":"ok"}}],"summary":{"ran_honours":2,"ran_draft_wrong":3,"ran":2,"unverified":5},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1},"listed":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"df697d2b038fd81b","entry":"_count_leading_whitespace","repo":"bminixhofer/alm","repo_kind":"listed","path":"tokenkit/align.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/align.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df697d2b038fd81b"}},{"code_sha256_prefix":"322a59478cf137c9","entry":"_get_increasing_offset_pairs","repo":"bminixhofer/alm","repo_kind":"listed","path":"tokenkit/align.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/align.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"322a59478cf137c9"}},{"code_sha256_prefix":"7599260f4b86d5cb","entry":"_get_left_count","repo":"bminixhofer/alm","repo_kind":"listed","path":"tokenkit/align.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/align.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7599260f4b86d5cb"}},{"code_sha256_prefix":"f88844e93e9833c0","entry":"_make_left_count_match_or_be_zero","repo":"bminixhofer/alm","repo_kind":"listed","path":"tokenkit/align.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/align.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f88844e93e9833c0"}},{"code_sha256_prefix":"2b0a0e9fa563dfe2","entry":"cosine_similarity","repo":"bminixhofer/zett","repo_kind":"official","path":"zett/utils.py","file_url":"https://github.com/bminixhofer/zett/blob/HEAD/zett/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2b0a0e9fa563dfe2"}},{"code_sha256_prefix":"5e0f65316506050d","entry":"huber_loss","repo":"bminixhofer/zett","repo_kind":"official","path":"zett/utils.py","file_url":"https://github.com/bminixhofer/zett/blob/HEAD/zett/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e0f65316506050d"}},{"code_sha256_prefix":"37423810abd3aed9","entry":"identity_byte_fn","repo":"bminixhofer/alm","repo_kind":"listed","path":"tokenkit/align.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/align.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37423810abd3aed9"}},{"code_sha256_prefix":"9e3479886595acee","entry":"BaseModelKind","repo":"bminixhofer/alm","repo_kind":"listed","path":"tokenkit/align.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/align.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9e3479886595acee"}},{"code_sha256_prefix":"af0c3aef5a8da60c","entry":"ByteifyTokenizer","repo":"bminixhofer/alm","repo_kind":"listed","path":"tokenkit/align.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/align.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af0c3aef5a8da60c"}},{"code_sha256_prefix":"f5449b81ca075977","entry":"get_alignment_indices","repo":"bminixhofer/alm","repo_kind":"listed","path":"tokenkit/align.py","file_url":"https://github.com/bminixhofer/alm/blob/HEAD/tokenkit/align.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f5449b81ca075977"}},{"code_sha256_prefix":"19bd359bd36dab7f","entry":"is_byte_level","repo":"bminixhofer/zett","repo_kind":"official","path":"zett/tokenizer_converters.py","file_url":"https://github.com/bminixhofer/zett/blob/HEAD/zett/tokenizer_converters.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"19bd359bd36dab7f"}},{"code_sha256_prefix":"adb54f0d1ae9c13b","entry":"softmax","repo":"bminixhofer/zett","repo_kind":"official","path":"zett/utils.py","file_url":"https://github.com/bminixhofer/zett/blob/HEAD/zett/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"adb54f0d1ae9c13b"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}