{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/humaneval-xl-a-multilingual-code-generation","title":"HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization","arxiv_id":"2402.16694","date":"2024-02-26","proceeding":null,"authors":["Qiwei Peng","Yekun Chai","Xuhong LI"],"abstract":"Large language models (LLMs) have made significant progress in generating codes from textual prompts. However, existing benchmarks have mainly concentrated on translating English prompts to multilingual codes or have been constrained to very limited natural languages (NLs). These benchmarks have overlooked the vast landscape of massively multilingual NL to multilingual code, leaving a critical gap in the evaluation of multilingual LLMs. In response, we introduce HumanEval-XL, a massively multilingual code generation benchmark specifically crafted to address this deficiency. HumanEval-XL establishes connections between 23 NLs and 12 programming languages (PLs), and comprises of a collection of 22,080 prompts with an average of 8.33 test cases. By ensuring parallel data across multiple NLs and PLs, HumanEval-XL offers a comprehensive evaluation platform for multilingual LLMs, allowing the assessment of the understanding of different NLs. Our work serves as a pioneering step towards filling the void in evaluating NL generalization in the area of multilingual code generation. We make our evaluation code and data publicly available at \\url{https://github.com/FloatAI/humaneval-xl}.","url_abs":"https://arxiv.org/abs/2402.16694v2","url_pdf":"https://arxiv.org/pdf/2402.16694v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"humaneval-xl-a-multilingual-code-generation","repo_url":"https://github.com/FloatAI/HumanEval-XL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"code-generation","task_name":"Code Generation"},{"task_slug":"humaneval","task_name":"HumanEval"}],"methods":[],"datasets_introduced":[{"slug":"humaneval-xl","name":"HumanEval-XL","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.16694","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.16694"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/FloatAI/HumanEval-XL","reach":null}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"5ecfb224042626be","entry":"check_correctness_java","repo":"FloatAI/HumanEval-XL","repo_kind":"official","path":"mxeval/mxeval/execution.py","file_url":"https://github.com/FloatAI/HumanEval-XL/blob/HEAD/mxeval/mxeval/execution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5ecfb224042626be"}},{"code_sha256_prefix":"52e5a8c7b9718201","entry":"check_correctness_perl","repo":"FloatAI/HumanEval-XL","repo_kind":"official","path":"mxeval/mxeval/execution.py","file_url":"https://github.com/FloatAI/HumanEval-XL/blob/HEAD/mxeval/mxeval/execution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"52e5a8c7b9718201"}},{"code_sha256_prefix":"442ebd35d651b935","entry":"check_correctness_scala","repo":"FloatAI/HumanEval-XL","repo_kind":"official","path":"mxeval/mxeval/execution.py","file_url":"https://github.com/FloatAI/HumanEval-XL/blob/HEAD/mxeval/mxeval/execution.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"442ebd35d651b935"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}