{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/extrapolating-large-language-models-to-non","title":"Extrapolating Large Language Models to Non-English by Aligning Languages","arxiv_id":"2308.04948","date":"2023-08-09","proceeding":null,"authors":["Wenhao Zhu","Yunzhe Lv","Qingxiu Dong","Fei Yuan","Jingjing Xu","ShuJian Huang","Lingpeng Kong","Jiajun Chen","Lei LI"],"abstract":"Existing large language models show disparate capability across different languages, due to the imbalance in the training data. Their performances on English tasks are often stronger than on tasks of other languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages. We start from targeting individual languages by performing cross-lingual instruction-tuning (CoIT) on LLaMA, i.e. tuning it with translation task data and cross-lingual general task data to obtain cross-lingual models (x-LLaMAs), and formulate underlying scaling laws to investigate the advantages of using scalable translation data. Then we perform multilingual instruction-tuning (MuIT) with mixed resources to build multilingual m-LLaMA. We also illustrate how we leverage the scaling laws to optimize data allocation in a resource-constrained setting. Experiment results on cross-lingual benchmarks XQUAD and MLQA show that x-LLaMAs surpass the English instruction-tuned counterpart (Alpaca) by an average of 27.83% across six non-English languages. Evaluation results on translation dataset Flores-101 show that x-LLaMAs outperform previous LLaMA-based models by an average of 18.89%. Encouragingly, m-LLaMA achieves comparable performance to x-LLaMAs on individual languages and demonstrates the ability to follow multilingual instructions. Further analysis on response content and representation space reveals the alignment of the multilingual semantic space within the middle layers of m-LLaMA.","url_abs":"https://arxiv.org/abs/2308.04948v2","url_pdf":"https://arxiv.org/pdf/2308.04948v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"extrapolating-large-language-models-to-non","repo_url":"https://github.com/owennju/x-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"extrapolating-large-language-models-to-non","repo_url":"https://github.com/NJUNLP/x-LLM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.04948","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.04948"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/owennju/x-llm","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/NJUNLP/x-LLM","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":3,"ran":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9b2dd0265fc6e6b6","entry":"encode_prompt","repo":"NJUNLP/x-LLM","repo_kind":"official","path":"generate_instruction.py","file_url":"https://github.com/NJUNLP/x-LLM/blob/HEAD/generate_instruction.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9b2dd0265fc6e6b6"}},{"code_sha256_prefix":"8ca6a5f9f77053c2","entry":"find_word_in_string","repo":"NJUNLP/x-LLM","repo_kind":"official","path":"generate_instruction.py","file_url":"https://github.com/NJUNLP/x-LLM/blob/HEAD/generate_instruction.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ca6a5f9f77053c2"}},{"code_sha256_prefix":"b934977219a4b0ec","entry":"gen_prompt","repo":"NJUNLP/x-LLM","repo_kind":"official","path":"eval/mieval.eval.py","file_url":"https://github.com/NJUNLP/x-LLM/blob/HEAD/eval/mieval.eval.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b934977219a4b0ec"}},{"code_sha256_prefix":"d07d04439cd1d44f","entry":"jload","repo":"NJUNLP/x-LLM","repo_kind":"official","path":"utils.py","file_url":"https://github.com/NJUNLP/x-LLM/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d07d04439cd1d44f"}},{"code_sha256_prefix":"a5e4ad4e11cf2143","entry":"post_process_gpt3_response","repo":"NJUNLP/x-LLM","repo_kind":"official","path":"generate_instruction.py","file_url":"https://github.com/NJUNLP/x-LLM/blob/HEAD/generate_instruction.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a5e4ad4e11cf2143"}},{"code_sha256_prefix":"57bd6a217c768db9","entry":"load_datasets","repo":"NJUNLP/x-LLM","repo_kind":"official","path":"utils.py","file_url":"https://github.com/NJUNLP/x-LLM/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"57bd6a217c768db9"}},{"code_sha256_prefix":"6e52ceed817396ba","entry":"query_gpt","repo":"NJUNLP/x-LLM","repo_kind":"official","path":"eval/mieval.eval.py","file_url":"https://github.com/NJUNLP/x-LLM/blob/HEAD/eval/mieval.eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6e52ceed817396ba"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}