{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ask-me-in-english-instead-cross-lingual","title":"Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries","arxiv_id":"2310.13132","date":"2023-10-19","proceeding":null,"authors":["Yiqiao Jin","Mohit Chandra","Gaurav Verma","Yibo Hu","Munmun De Choudhury","Srijan Kumar"],"abstract":"Large language models (LLMs) are transforming the ways the general public accesses and consumes information. Their influence is particularly pronounced in pivotal sectors like healthcare, where lay individuals are increasingly appropriating LLMs as conversational agents for everyday queries. While LLMs demonstrate impressive language understanding and generation proficiencies, concerns regarding their safety remain paramount in these high-stake domains. Moreover, the development of LLMs is disproportionately focused on English. It remains unclear how these LLMs perform in the context of non-English languages, a gap that is critical for ensuring equity in the real-world use of these systems.This paper provides a framework to investigate the effectiveness of LLMs as multi-lingual dialogue systems for healthcare queries. Our empirically-derived framework XlingEval focuses on three fundamental criteria for evaluating LLM responses to naturalistic human-authored health-related questions: correctness, consistency, and verifiability. Through extensive experiments on four major global languages, including English, Spanish, Chinese, and Hindi, spanning three expert-annotated large health Q&A datasets, and through an amalgamation of algorithmic and human-evaluation strategies, we found a pronounced disparity in LLM responses across these languages, indicating a need for enhanced cross-lingual capabilities. We further propose XlingHealth, a cross-lingual benchmark for examining the multilingual capabilities of LLMs in the healthcare context. Our findings underscore the pressing need to bolster the cross-lingual capacities of these models, and to provide an equitable information ecosystem accessible to all.","url_abs":"https://arxiv.org/abs/2310.13132v2","url_pdf":"https://arxiv.org/pdf/2310.13132v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ask-me-in-english-instead-cross-lingual","repo_url":"https://github.com/claws-lab/XLingEval","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[{"slug":"xlingeval","name":"XLingEval","full_name":"XLingEval"}],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2310.13132","atlas_url":"https://app.syntology.ai/?focus=2310.13132","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.13132"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/claws-lab/XLingEval","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":8,"unverified":2},"by_repo_kind":{"official":{"samples":10,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b4bf5a8362f1b8a1","entry":"format_question","repo":"claws-lab/XLingEval","repo_kind":"official","path":"consistency/consistency_get_medalpaca_answer.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/consistency/consistency_get_medalpaca_answer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b4bf5a8362f1b8a1"}},{"code_sha256_prefix":"862b92b43e3f1274","entry":"get_consistency_results_path","repo":"claws-lab/XLingEval","repo_kind":"official","path":"consistency/data_consistency.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/consistency/data_consistency.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"862b92b43e3f1274"}},{"code_sha256_prefix":"5c71aa86a1321285","entry":"get_topic_mean_std","repo":"claws-lab/XLingEval","repo_kind":"official","path":"consistency/consistency_answer_evaluation.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/consistency/consistency_answer_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5c71aa86a1321285"}},{"code_sha256_prefix":"9a52e4e9f92b3c06","entry":"jaccard_similarity","repo":"claws-lab/XLingEval","repo_kind":"official","path":"consistency/consistency_answer_evaluation.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/consistency/consistency_answer_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a52e4e9f92b3c06"}},{"code_sha256_prefix":"89fa36f811db1e8c","entry":"prompt_consistency","repo":"claws-lab/XLingEval","repo_kind":"official","path":"consistency/prompts.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/consistency/prompts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"89fa36f811db1e8c"}},{"code_sha256_prefix":"6f941b073ff9115d","entry":"prompt_type","repo":"claws-lab/XLingEval","repo_kind":"official","path":"correctness/correctness_answer_evaluation.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/correctness/correctness_answer_evaluation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6f941b073ff9115d"}},{"code_sha256_prefix":"19437fe46b756fdc","entry":"starts_with_capital_letter","repo":"claws-lab/XLingEval","repo_kind":"official","path":"consistency/consistency_get_medalpaca_answer.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/consistency/consistency_get_medalpaca_answer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"19437fe46b756fdc"}},{"code_sha256_prefix":"fc353be025870298","entry":"strip_special_chars","repo":"claws-lab/XLingEval","repo_kind":"official","path":"consistency/consistency_get_medalpaca_answer.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/consistency/consistency_get_medalpaca_answer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fc353be025870298"}},{"code_sha256_prefix":"94dd8c74f64544e7","entry":"load_results_consistency","repo":"claws-lab/XLingEval","repo_kind":"official","path":"consistency/data_consistency.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/consistency/data_consistency.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94dd8c74f64544e7"}},{"code_sha256_prefix":"4b4485552b993f41","entry":"split_multilingual_paragraph","repo":"claws-lab/XLingEval","repo_kind":"official","path":"consistency/language_consistency.py","file_url":"https://github.com/claws-lab/XLingEval/blob/HEAD/consistency/language_consistency.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b4485552b993f41"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}