{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tydi-qa-a-benchmark-for-information-seeking","title":"TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages","arxiv_id":"2003.05002","date":"2020-03-10","proceeding":"TACL 2020 1","authors":["Jonathan H. Clark","Eunsol Choi","Michael Collins","Dan Garrette","Tom Kwiatkowski","Vitaly Nikolaev","Jennimaria Palomaki"],"abstract":"Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TyDi QA---a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology---the set of linguistic features each language expresses---such that we expect models performing well on this set to generalize across a large number of the world's languages. We present a quantitative analysis of the data quality and example-level qualitative linguistic analyses of observed language phenomena that would not be found in English-only corpora. To provide a realistic information-seeking task and avoid priming effects, questions are written by people who want to know the answer, but don't know the answer yet, and the data is collected directly in each language without the use of translation.","url_abs":"https://arxiv.org/abs/2003.05002v1","url_pdf":"https://arxiv.org/pdf/2003.05002v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tydi-qa-a-benchmark-for-information-seeking","repo_url":"https://github.com/google-research-datasets/tydiqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null},{"paper_slug":"tydi-qa-a-benchmark-for-information-seeking","repo_url":"https://github.com/castorini/mr.tydi","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"translation","task_name":"Translation"}],"methods":[],"datasets_introduced":[{"slug":"tydi-qa","name":"TyDiQA","full_name":"Typologically Diverse Question Answering"},{"slug":"tydiqa-goldp","name":"TyDiQA-GoldP","full_name":"TyDiQA-GoldP"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2003.05002","atlas_url":"https://app.syntology.ai/?focus=2003.05002","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2003.05002"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/castorini/mr.tydi","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research-datasets/tydiqa","reach":null}],"summary":{"ran_draft_wrong":2},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"17de590bd107cded","entry":"byte_slice","repo":"google-research-datasets/tydiqa","repo_kind":"official","path":"tydi_eval.py","file_url":"https://github.com/google-research-datasets/tydiqa/blob/HEAD/tydi_eval.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17de590bd107cded"}},{"code_sha256_prefix":"8735894f4e3d26ec","entry":"load_runs","repo":"castorini/mr.tydi","repo_kind":"listed","path":"scripts/hybrid.py","file_url":"https://github.com/castorini/mr.tydi/blob/HEAD/scripts/hybrid.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8735894f4e3d26ec"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}