{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mexa-multilingual-evaluation-of-english","title":"MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment","arxiv_id":"2410.05873","date":"2024-10-08","proceeding":null,"authors":["Amir Hossein Kargaran","Ali Modarressi","Nafiseh Nikeghbal","Jana Diesner","François Yvon","Hinrich Schütze"],"abstract":"English-centric large language models (LLMs) often show strong multilingual capabilities. However, the multilingual performance of these models remains unclear and is not thoroughly evaluated for many languages. Most benchmarks for multilinguality focus on classic NLP tasks, or cover a minimal number of languages. We introduce MEXA, a method for assessing the multilingual capabilities of pre-trained English-centric LLMs using parallel sentences, which are available for more languages than existing downstream tasks. MEXA leverages the fact that English-centric LLMs use English as a kind of pivot language in their intermediate layers. It computes the alignment between English and non-English languages using parallel sentences to evaluate the transfer of language understanding from English to other languages. This alignment can be used to estimate model performance in other languages. We conduct studies using various parallel datasets (FLORES-200 and Bible), models (Llama family, Gemma family, Mistral, and OLMo), and established downstream tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute embeddings in decoder-only models. Our results show that MEXA, in its default settings, achieves a statistically significant average Pearson correlation of 0.90 with three established downstream tasks across nine models and two parallel datasets. This suggests that MEXA is a reliable method for estimating the multilingual capabilities of English-centric LLMs, providing a clearer understanding of their multilingual potential and the inner workings of LLMs. Leaderboard: https://huggingface.co/spaces/cis-lmu/Mexa, Code: https://github.com/cisnlp/Mexa.","url_abs":"https://arxiv.org/abs/2410.05873v1","url_pdf":"https://arxiv.org/pdf/2410.05873v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mexa-multilingual-evaluation-of-english","repo_url":"https://github.com/cisnlp/mexa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"arc","task_name":"ARC"},{"task_slug":"belebele","task_name":"Belebele"},{"task_slug":"mmlu","task_name":"MMLU"}],"methods":[],"datasets_introduced":[{"slug":"spbc","name":"sPBC","full_name":"Super Parallel Bible Corpus"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2410.05873","atlas_url":"https://app.syntology.ai/?focus=2410.05873","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.05873"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cisnlp/mexa","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_fixture":2,"ran_honours":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":6,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"dbe1b491ed11294e","entry":"cosine_similarity","repo":"cisnlp/mexa","repo_kind":"official","path":"compute_mexa.py","file_url":"https://github.com/cisnlp/mexa/blob/HEAD/compute_mexa.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dbe1b491ed11294e"}},{"code_sha256_prefix":"4f8f7ab44490ff31","entry":"lasttoken_embeddings","repo":"cisnlp/mexa","repo_kind":"official","path":"embed_extractor.py","file_url":"https://github.com/cisnlp/mexa/blob/HEAD/embed_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4f8f7ab44490ff31"}},{"code_sha256_prefix":"54d36012f8d16acc","entry":"mexa","repo":"cisnlp/mexa","repo_kind":"official","path":"compute_mexa.py","file_url":"https://github.com/cisnlp/mexa/blob/HEAD/compute_mexa.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"54d36012f8d16acc"}},{"code_sha256_prefix":"dfb4ec39afc055bb","entry":"weighted_embeddings","repo":"cisnlp/mexa","repo_kind":"official","path":"embed_extractor.py","file_url":"https://github.com/cisnlp/mexa/blob/HEAD/embed_extractor.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dfb4ec39afc055bb"}},{"code_sha256_prefix":"ba6588b1c55fb556","entry":"compute_distance","repo":"cisnlp/mexa","repo_kind":"official","path":"compute_mexa.py","file_url":"https://github.com/cisnlp/mexa/blob/HEAD/compute_mexa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ba6588b1c55fb556"}},{"code_sha256_prefix":"1e14cc3a1c57ab56","entry":"get_embedding_layers","repo":"cisnlp/mexa","repo_kind":"official","path":"embed_extractor.py","file_url":"https://github.com/cisnlp/mexa/blob/HEAD/embed_extractor.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e14cc3a1c57ab56"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}