{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/do-llamas-work-in-english-on-the-latent","title":"Do Llamas Work in English? On the Latent Language of Multilingual Transformers","arxiv_id":"2402.10588","date":"2024-02-16","proceeding":null,"authors":["Chris Wendler","Veniamin Veselovsky","Giovanni Monea","Robert West"],"abstract":"We ask whether multilingual language models trained on unbalanced, English-dominated corpora use English as an internal pivot language -- a question of key importance for understanding how language models function and the origins of linguistic bias. Focusing on the Llama-2 family of transformer models, our study uses carefully constructed non-English prompts with a unique correct single-token continuation. From layer to layer, transformers gradually map an input embedding of the final prompt token to an output embedding from which next-token probabilities are computed. Tracking intermediate embeddings through their high-dimensional space reveals three distinct phases, whereby intermediate embeddings (1) start far away from output token embeddings; (2) already allow for decoding a semantically correct next token in the middle layers, but give higher probability to its version in English than in the input language; (3) finally move into an input-language-specific region of the embedding space. We cast these results into a conceptual model where the three phases operate in \"input space\", \"concept space\", and \"output space\", respectively. Crucially, our evidence suggests that the abstract \"concept space\" lies closer to English than to other languages, which may have important consequences regarding the biases held by multilingual language models.","url_abs":"https://arxiv.org/abs/2402.10588v4","url_pdf":"https://arxiv.org/pdf/2402.10588v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"do-llamas-work-in-english-on-the-latent","repo_url":"https://github.com/epfl-dlab/llm-latent-language","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.10588","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.10588"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/epfl-dlab/llm-latent-language","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"59d98037d0355da4","entry":"load_pickle","repo":"epfl-dlab/llm-latent-language","repo_kind":"official","path":"utils.py","file_url":"https://github.com/epfl-dlab/llm-latent-language/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59d98037d0355da4"}},{"code_sha256_prefix":"2ab9ad6901869028","entry":"plot_ci_plus_heatmap","repo":"epfl-dlab/llm-latent-language","repo_kind":"official","path":"utils.py","file_url":"https://github.com/epfl-dlab/llm-latent-language/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ab9ad6901869028"}},{"code_sha256_prefix":"0557332cc65ac319","entry":"yaml_to_dict","repo":"epfl-dlab/llm-latent-language","repo_kind":"official","path":"utils.py","file_url":"https://github.com/epfl-dlab/llm-latent-language/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0557332cc65ac319"}},{"code_sha256_prefix":"aa16d4a0d7373a3a","entry":"load_tokenizer_only","repo":"epfl-dlab/llm-latent-language","repo_kind":"official","path":"llamawrapper.py","file_url":"https://github.com/epfl-dlab/llm-latent-language/blob/HEAD/llamawrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa16d4a0d7373a3a"}},{"code_sha256_prefix":"990785dbe8cf6698","entry":"load_unemb_only","repo":"epfl-dlab/llm-latent-language","repo_kind":"official","path":"llamawrapper.py","file_url":"https://github.com/epfl-dlab/llm-latent-language/blob/HEAD/llamawrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"990785dbe8cf6698"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}