{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mlqa-evaluating-cross-lingual-extractive","title":"MLQA: Evaluating Cross-lingual Extractive Question Answering","arxiv_id":"1910.07475","date":"2019-10-16","proceeding":"ACL 2020 6","authors":["Patrick Lewis","Barlas Oğuz","Ruty Rinott","Sebastian Riedel","Holger Schwenk"],"abstract":"Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. Such annotated datasets are difficult and costly to collect, and rarely exist in languages other than English, making training QA systems in other languages challenging. An alternative to building large monolingual training datasets is to develop cross-lingual systems which can transfer to a target language without requiring training data in that language. In order to develop such systems, it is crucial to invest in high quality multilingual evaluation benchmarks to measure progress. We present MLQA, a multi-way aligned extractive QA evaluation benchmark intended to spur research in this area. MLQA contains QA instances in 7 languages, namely English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified Chinese. It consists of over 12K QA instances in English and 5K in each other language, with each QA instance being parallel between 4 languages on average. MLQA is built using a novel alignment context strategy on Wikipedia articles, and serves as a cross-lingual extension to existing extractive QA datasets. We evaluate current state-of-the-art cross-lingual representations on MLQA, and also provide machine-translation-based baselines. In all cases, transfer results are shown to be significantly behind training-language performance.","url_abs":"https://arxiv.org/abs/1910.07475v3","url_pdf":"https://arxiv.org/pdf/1910.07475v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mlqa-evaluating-cross-lingual-extractive","repo_url":"https://github.com/facebookresearch/MLQA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"mlqa-evaluating-cross-lingual-extractive","repo_url":"https://github.com/ccasimiro88/TranslateAlignRetrieve","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"mlqa-evaluating-cross-lingual-extractive","repo_url":"https://github.com/lmarent/TranslateAlignRetrieve","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"mlqa-evaluating-cross-lingual-extractive","repo_url":"https://github.com/stonybrooknlp/musique","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"extractive-question-answering","task_name":"Extractive Question-Answering"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[],"datasets_introduced":[{"slug":"mlqa","name":"MLQA","full_name":"MultiLingual Question Answering"}],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1910.07475","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.07475"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ccasimiro88/TranslateAlignRetrieve","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/facebookresearch/MLQA","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lmarent/TranslateAlignRetrieve","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stonybrooknlp/musique","reach":null}],"summary":{"ran_draft_wrong":3},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"b6a95b947b35a4d6","entry":"mixed_segmentation","repo":"facebookresearch/MLQA","repo_kind":"official","path":"mlqa_evaluation_v1.py","file_url":"https://github.com/facebookresearch/MLQA/blob/HEAD/mlqa_evaluation_v1.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b6a95b947b35a4d6"}},{"code_sha256_prefix":"9cddf0b1a2750978","entry":"normalize_answer","repo":"facebookresearch/MLQA","repo_kind":"official","path":"mlqa_evaluation_v1.py","file_url":"https://github.com/facebookresearch/MLQA/blob/HEAD/mlqa_evaluation_v1.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9cddf0b1a2750978"}},{"code_sha256_prefix":"a5f7f44e35526909","entry":"whitespace_tokenize","repo":"facebookresearch/MLQA","repo_kind":"official","path":"mlqa_evaluation_v1.py","file_url":"https://github.com/facebookresearch/MLQA/blob/HEAD/mlqa_evaluation_v1.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a5f7f44e35526909"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}