{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-belebele-benchmark-a-parallel-reading","title":"The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants","arxiv_id":"2308.16884","date":"2023-08-31","proceeding":null,"authors":["Lucas Bandarkar","Davis Liang","Benjamin Muller","Mikel Artetxe","Satya Narayan Shukla","Donald Husa","Naman Goyal","Abhinandan Krishnan","Luke Zettlemoyer","Madian Khabsa"],"abstract":"We present Belebele, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset enables the evaluation of text models in high-, medium-, and low-resource languages. Each question is based on a short passage from the Flores-200 dataset and has four multiple-choice answers. The questions were carefully curated to discriminate between models with different levels of general language comprehension. The English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. We use this dataset to evaluate the capabilities of multilingual masked language models (MLMs) and large language models (LLMs). We present extensive results and find that despite significant cross-lingual transfer in English-centric LLMs, much smaller MLMs pretrained on balanced multilingual data still understand far more languages. We also observe that larger vocabulary size and conscious vocabulary construction correlate with better performance on low-resource languages. Overall, Belebele opens up new avenues for evaluating and analyzing the multilingual capabilities of NLP systems.","url_abs":"https://arxiv.org/abs/2308.16884v2","url_pdf":"https://arxiv.org/pdf/2308.16884v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-belebele-benchmark-a-parallel-reading","repo_url":"https://github.com/facebookresearch/belebele","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"the-belebele-benchmark-a-parallel-reading","repo_url":"https://github.com/BunsenFeng/M-AbstainQA","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"belebele","task_name":"Belebele"},{"task_slug":"cross-lingual-transfer","task_name":"Cross-Lingual Transfer"},{"task_slug":"machine-reading-comprehension","task_name":"Machine Reading Comprehension"},{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"reading-comprehension","task_name":"Reading Comprehension"}],"methods":[],"datasets_introduced":[{"slug":"belebele","name":"Belebele","full_name":""}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2308.16884","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.16884"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/belebele","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/BunsenFeng/M-AbstainQA","reach":{"status":"ok"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"0f240295cda5f225","entry":"download_files","repo":"facebookresearch/belebele","repo_kind":"official","path":"assemble_training_set.py","file_url":"https://github.com/facebookresearch/belebele/blob/HEAD/assemble_training_set.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0f240295cda5f225"}},{"code_sha256_prefix":"906cb4bb71281209","entry":"load_fleurs","repo":"facebookresearch/belebele","repo_kind":"official","path":"2M-belebele/align/align.py","file_url":"https://github.com/facebookresearch/belebele/blob/HEAD/2M-belebele/align/align.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"906cb4bb71281209"}},{"code_sha256_prefix":"67d7d3950564f440","entry":"load_flores_ids","repo":"facebookresearch/belebele","repo_kind":"official","path":"2M-belebele/align/align.py","file_url":"https://github.com/facebookresearch/belebele/blob/HEAD/2M-belebele/align/align.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"67d7d3950564f440"}},{"code_sha256_prefix":"cd474bb69dba862d","entry":"load_flores_urls","repo":"facebookresearch/belebele","repo_kind":"official","path":"2M-belebele/align/align.py","file_url":"https://github.com/facebookresearch/belebele/blob/HEAD/2M-belebele/align/align.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cd474bb69dba862d"}},{"code_sha256_prefix":"678c462c6c83e408","entry":"process_multirc","repo":"facebookresearch/belebele","repo_kind":"official","path":"assemble_training_set.py","file_url":"https://github.com/facebookresearch/belebele/blob/HEAD/assemble_training_set.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"678c462c6c83e408"}},{"code_sha256_prefix":"8f4832aa36276af5","entry":"process_sciq","repo":"facebookresearch/belebele","repo_kind":"official","path":"assemble_training_set.py","file_url":"https://github.com/facebookresearch/belebele/blob/HEAD/assemble_training_set.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8f4832aa36276af5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}