{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/how-do-large-language-models-handle","title":"How do Large Language Models Handle Multilingualism?","arxiv_id":"2402.18815","date":"2024-02-29","proceeding":null,"authors":["Yiran Zhao","Wenxuan Zhang","Guizhen Chen","Kenji Kawaguchi","Lidong Bing"],"abstract":"Large language models (LLMs) have demonstrated impressive capabilities across diverse languages. This study explores how LLMs handle multilingualism. Based on observed language ratio shifts among layers and the relationships between network structures and certain capabilities, we hypothesize the LLM's multilingual workflow ($\\texttt{MWork}$): LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures, respectively. In the final layers, LLMs generate responses aligned with the original language of the query. To verify $\\texttt{MWork}$, we introduce Parallel Language-specific Neuron Detection ($\\texttt{PLND}$) to identify activated neurons for inputs in different languages without any labeled data. Using $\\texttt{PLND}$, we validate $\\texttt{MWork}$ through extensive experiments involving the deactivation of language-specific neurons across various layers and structures. Moreover, $\\texttt{MWork}$ allows fine-tuning of language-specific neurons with a small dataset, enhancing multilingual abilities in a specific language without compromising others. This approach results in an average improvement of $3.6\\%$ for high-resource languages and $2.3\\%$ for low-resource languages across all tasks with just $400$ documents.","url_abs":"https://arxiv.org/abs/2402.18815v3","url_pdf":"https://arxiv.org/pdf/2402.18815v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"how-do-large-language-models-handle","repo_url":"https://github.com/damo-nlp-sg/multilingual_analysis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"align","method_name":"ALIGN"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.18815","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.18815"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/damo-nlp-sg/multilingual_analysis","reach":{"status":"ok"}}],"summary":{"ran_fixture":2,"ran_draft_wrong":2,"ran":5,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"layers/transformers/models/modeling_llama.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/layers/transformers/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"d61c483a3c2b3156","entry":"apply_rotary_pos_emb","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"layers/transformers/models/modeling_qwen2.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/layers/transformers/models/modeling_qwen2.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d61c483a3c2b3156"}},{"code_sha256_prefix":"bac65c3dafaec040","entry":"apply_rotary_pos_emb","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"layers/transformers/models/modeling_llama.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/layers/transformers/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bac65c3dafaec040"}},{"code_sha256_prefix":"fea555fa29409762","entry":"bloom_gelu_forward","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"layers/transformers/models/modeling_bloom.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/layers/transformers/models/modeling_bloom.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fea555fa29409762"}},{"code_sha256_prefix":"4a678475b134e3a3","entry":"build_alibi_tensor","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"layers/transformers/models/modeling_bloom.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/layers/transformers/models/modeling_bloom.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4a678475b134e3a3"}},{"code_sha256_prefix":"351bf96fa5d82bb3","entry":"deduplicate","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"neuron_enhancement/train_neuron.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/neuron_enhancement/train_neuron.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"351bf96fa5d82bb3"}},{"code_sha256_prefix":"546c22d62993a619","entry":"dropout_add","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"layers/transformers/models/modeling_bloom.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/layers/transformers/models/modeling_bloom.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"546c22d62993a619"}},{"code_sha256_prefix":"ef2b510b7d7a78bb","entry":"formatting_prompts_func","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"neuron_enhancement/train_neuron.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/neuron_enhancement/train_neuron.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ef2b510b7d7a78bb"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"layers/transformers/models/modeling_llama.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/layers/transformers/models/modeling_llama.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"eec474cc8cda034e","entry":"top_k_top_p_filtering","repo":"damo-nlp-sg/multilingual_analysis","repo_kind":"official","path":"layers/transformers/generation/utils.py","file_url":"https://github.com/damo-nlp-sg/multilingual_analysis/blob/HEAD/layers/transformers/generation/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eec474cc8cda034e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}