{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/vocabulary-expansion-of-chat-models-with","title":"Vocabulary Expansion of Chat Models with Unlabeled Target Language Data","arxiv_id":"2412.11704","date":"2024-12-16","proceeding":null,"authors":["Atsuki Yamaguchi","Terufumi Morishita","Aline Villavicencio","Nikolaos Aletras"],"abstract":"Chat models (i.e. language models trained to follow instructions through conversation with humans) outperform base models (i.e. trained solely on unlabeled data) in both conversation and general task-solving abilities. These models are generally English-centric and require further adaptation for languages that are underrepresented in or absent from their training data. A common technique for adapting base models is to extend the model's vocabulary with target language tokens, i.e. vocabulary expansion (VE), and then continually pre-train it on language-specific data. Using chat data is ideal for chat model adaptation, but often, either this does not exist or is costly to construct. Alternatively, adapting chat models with unlabeled data is a possible solution, but it could result in catastrophic forgetting. In this paper, we investigate the impact of using unlabeled target language data for VE on chat models for the first time. We first show that off-the-shelf VE generally performs well across target language tasks and models in 71% of cases, though it underperforms in scenarios where source chat models are already strong. To further improve adapted models, we propose post-hoc techniques that inject information from the source model without requiring any further training. Experiments reveal the effectiveness of our methods, helping the adapted models to achieve performance improvements in 87% of cases.","url_abs":"https://arxiv.org/abs/2412.11704v2","url_pdf":"https://arxiv.org/pdf/2412.11704v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"vocabulary-expansion-of-chat-models-with","repo_url":"https://github.com/gucci-j/chat-cve","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"base","method_name":"BASE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2412.11704","atlas_url":"https://app.syntology.ai/?focus=2412.11704","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2412.11704"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gucci-j/chat-cve","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7a9ade35bc5412f1","entry":"add_transition","repo":"gucci-j/chat-cve","repo_kind":"official","path":"merging/src/methods/add_transition.py","file_url":"https://github.com/gucci-j/chat-cve/blob/HEAD/merging/src/methods/add_transition.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a9ade35bc5412f1"}},{"code_sha256_prefix":"6d1ce1df079d0080","entry":"apply_swapping","repo":"gucci-j/chat-cve","repo_kind":"official","path":"merging/src/methods/apply_swapping.py","file_url":"https://github.com/gucci-j/chat-cve/blob/HEAD/merging/src/methods/apply_swapping.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d1ce1df079d0080"}},{"code_sha256_prefix":"b0ae80345791e4ad","entry":"copy_emb","repo":"gucci-j/chat-cve","repo_kind":"official","path":"merging/src/methods/copy_emb.py","file_url":"https://github.com/gucci-j/chat-cve/blob/HEAD/merging/src/methods/copy_emb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0ae80345791e4ad"}},{"code_sha256_prefix":"48cc29d2c1c88814","entry":"instantiate_model_by_mean","repo":"gucci-j/chat-cve","repo_kind":"official","path":"instantiation/src/methods/mean.py","file_url":"https://github.com/gucci-j/chat-cve/blob/HEAD/instantiation/src/methods/mean.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"48cc29d2c1c88814"}},{"code_sha256_prefix":"7909323ec4e084e4","entry":"lang_code_to_instruction","repo":"gucci-j/chat-cve","repo_kind":"official","path":"evaluation/src/gmmlu.py","file_url":"https://github.com/gucci-j/chat-cve/blob/HEAD/evaluation/src/gmmlu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7909323ec4e084e4"}},{"code_sha256_prefix":"de8aed064593dbc8","entry":"process_judge_response","repo":"gucci-j/chat-cve","repo_kind":"official","path":"evaluation/src/mtbench.py","file_url":"https://github.com/gucci-j/chat-cve/blob/HEAD/evaluation/src/mtbench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"de8aed064593dbc8"}},{"code_sha256_prefix":"cee6083e09139c9a","entry":"round_to_nearest_multiple","repo":"gucci-j/chat-cve","repo_kind":"official","path":"instantiation/src/methods/mean.py","file_url":"https://github.com/gucci-j/chat-cve/blob/HEAD/instantiation/src/methods/mean.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cee6083e09139c9a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}