{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/an-efficient-multilingual-language-model","title":"An Efficient Multilingual Language Model Compression through Vocabulary Trimming","arxiv_id":"2305.15020","date":"2023-05-24","proceeding":null,"authors":["Asahi Ushio","Yi Zhou","Jose Camacho-Collados"],"abstract":"Multilingual language model (LM) have become a powerful tool in NLP especially for non-English languages. Nevertheless, model parameters of multilingual LMs remain large due to the larger embedding matrix of the vocabulary covering tokens in different languages. On the contrary, monolingual LMs can be trained in a target language with the language-specific vocabulary only, but this requires a large budget and availability of reliable corpora to achieve a high-quality LM from scratch. In this paper, we propose vocabulary-trimming (VT), a method to reduce a multilingual LM vocabulary to a target language by deleting irrelevant tokens from its vocabulary. In theory, VT can compress any existing multilingual LM to build monolingual LMs in any language covered by the multilingual LM. In our experiments, we show that VT can retain the original performance of the multilingual LM, while being smaller in size (in general around 50% of the original vocabulary size is enough) than the original multilingual LM. The evaluation is performed over four NLP tasks (two generative and two classification tasks) among four widely used multilingual LMs in seven languages. Finally, we show that this methodology can keep the best of both monolingual and multilingual worlds by keeping a small size as monolingual models without the need for specifically retraining them, and even limiting potentially harmful social biases.","url_abs":"https://arxiv.org/abs/2305.15020v3","url_pdf":"https://arxiv.org/pdf/2305.15020v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"an-efficient-multilingual-language-model","repo_url":"https://github.com/asahi417/lm-vocab-trimmer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"model-compression","task_name":"Model Compression"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2305.15020","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2305.15020"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/asahi417/lm-vocab-trimmer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"cf86152564d8753e","entry":"download","repo":"asahi417/lm-vocab-trimmer","repo_kind":"official","path":"experiments/result.qa_qg.py","file_url":"https://github.com/asahi417/lm-vocab-trimmer/blob/HEAD/experiments/result.qa_qg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf86152564d8753e"}},{"code_sha256_prefix":"6a3affbd85c7c0bb","entry":"download","repo":"asahi417/lm-vocab-trimmer","repo_kind":"official","path":"experiments/result.sentiment_xnli.py","file_url":"https://github.com/asahi417/lm-vocab-trimmer/blob/HEAD/experiments/result.sentiment_xnli.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a3affbd85c7c0bb"}},{"code_sha256_prefix":"e258efb71ebab463","entry":"emphasize","repo":"asahi417/lm-vocab-trimmer","repo_kind":"official","path":"experiments/result.qa_qg.py","file_url":"https://github.com/asahi417/lm-vocab-trimmer/blob/HEAD/experiments/result.qa_qg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e258efb71ebab463"}},{"code_sha256_prefix":"6f04b0fee3a05006","entry":"get_cache_dir","repo":"asahi417/lm-vocab-trimmer","repo_kind":"official","path":"vocabtrimmer/util.py","file_url":"https://github.com/asahi417/lm-vocab-trimmer/blob/HEAD/vocabtrimmer/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6f04b0fee3a05006"}},{"code_sha256_prefix":"337714394b87af04","entry":"pretty","repo":"asahi417/lm-vocab-trimmer","repo_kind":"official","path":"vocabtrimmer/util.py","file_url":"https://github.com/asahi417/lm-vocab-trimmer/blob/HEAD/vocabtrimmer/util.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"337714394b87af04"}},{"code_sha256_prefix":"e607de1f62f6d877","entry":"show_parameter","repo":"asahi417/lm-vocab-trimmer","repo_kind":"official","path":"vocabtrimmer/base_trimmer.py","file_url":"https://github.com/asahi417/lm-vocab-trimmer/blob/HEAD/vocabtrimmer/base_trimmer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e607de1f62f6d877"}},{"code_sha256_prefix":"118c134ef85e29c4","entry":"show_table","repo":"asahi417/lm-vocab-trimmer","repo_kind":"official","path":"experiments/result.qa_qg.py","file_url":"https://github.com/asahi417/lm-vocab-trimmer/blob/HEAD/experiments/result.qa_qg.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"118c134ef85e29c4"}},{"code_sha256_prefix":"80a9c43061429b76","entry":"update_fq","repo":"asahi417/lm-vocab-trimmer","repo_kind":"official","path":"vocabtrimmer/vocab_miner.py","file_url":"https://github.com/asahi417/lm-vocab-trimmer/blob/HEAD/vocabtrimmer/vocab_miner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80a9c43061429b76"}},{"code_sha256_prefix":"b3cfed138086ef7e","entry":"vocab_miner","repo":"asahi417/lm-vocab-trimmer","repo_kind":"official","path":"vocabtrimmer/vocab_miner.py","file_url":"https://github.com/asahi417/lm-vocab-trimmer/blob/HEAD/vocabtrimmer/vocab_miner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b3cfed138086ef7e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}