{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2512-11437","title":"CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcare","arxiv_id":"2512.11437","date":"2025-12-12","proceeding":null,"authors":["Akash Ghosh","Srivarshinee Sridhar","Raghav Kaushik Ravi","Muhsin Muhsin","Sriparna Saha","Chirag Agarwal"],"abstract":"Integrating language models (LMs) in healthcare systems holds great promise for improving medical workflows and decision-making. However, a critical barrier to their real-world adoption is the lack of reliable evaluation of their trustworthiness, especially in multilingual healthcare settings. Existing LMs are predominantly trained in high-resource languages, making them ill-equipped to handle the complexity and diversity of healthcare queries in mid- and low-resource languages, posing significant challenges for deploying them in global healthcare contexts where linguistic diversity is key. In this work, we present CLINIC, a Comprehensive Multilingual Benchmark to evaluate the trustworthiness of language models in healthcare. CLINIC systematically benchmarks LMs across five key dimensions of trustworthiness: truthfulness, fairness, safety, robustness, and privacy, operationalized through 18 diverse tasks, spanning 15 languages (covering all the major continents), and encompassing a wide array of critical healthcare topics like disease conditions, preventive actions, diagnostic tests, treatments, surgeries, and medications. Our extensive evaluation reveals that LMs struggle with factual correctness, demonstrate bias across demographic and linguistic groups, and are susceptible to privacy breaches and adversarial attacks. By highlighting these shortcomings, CLINIC lays the foundation for enhancing the global reach and safety of LMs in healthcare across diverse languages.","url_abs":"https://arxiv.org/abs/2512.11437","url_pdf":"https://arxiv.org/pdf/2512.11437","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2512.11437","atlas_url":"https://app.syntology.ai/?focus=2512.11437","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2512.11437"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/AikyamLab/clinic","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":16},"by_repo_kind":{"found_in_text":{"samples":16,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0bc2a2920d72eb2b","entry":"clean_and_extract_choice","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/colloquial/colloquial_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/colloquial/colloquial_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bc2a2920d72eb2b"}},{"code_sha256_prefix":"eee9780706ca91ac","entry":"clean_response","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/adverserial/adv_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/adverserial/adv_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eee9780706ca91ac"}},{"code_sha256_prefix":"1b68e4a2fe2400a0","entry":"clean_response","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/consistency/consistency_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/consistency/consistency_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1b68e4a2fe2400a0"}},{"code_sha256_prefix":"d3064a9f7b1e0b34","entry":"clean_response","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/disparagement/disp_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/disparagement/disp_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3064a9f7b1e0b34"}},{"code_sha256_prefix":"d99cb6304d112c48","entry":"clean_response","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/exaggerated_safety/exagsaf.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/exaggerated_safety/exagsaf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d99cb6304d112c48"}},{"code_sha256_prefix":"e51b60b9add827b6","entry":"get_embeddings","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/adverserial/adv_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/adverserial/adv_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e51b60b9add827b6"}},{"code_sha256_prefix":"daf8bacd15f6bdea","entry":"get_embeddings","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/consistency/consistency_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/consistency/consistency_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"daf8bacd15f6bdea"}},{"code_sha256_prefix":"7fdf75bfa127d457","entry":"get_score","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/colloquial/colloquial_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/colloquial/colloquial_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7fdf75bfa127d457"}},{"code_sha256_prefix":"a5fb7b464744eb20","entry":"get_score","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/disparagement/disp_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/disparagement/disp_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5fb7b464744eb20"}},{"code_sha256_prefix":"987ba214da993053","entry":"get_score","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/exaggerated_safety/exagsaf.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/exaggerated_safety/exagsaf.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"987ba214da993053"}},{"code_sha256_prefix":"018247f0e18ff6f9","entry":"get_score_fct","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/hallucinations/hallu_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/hallucinations/hallu_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"018247f0e18ff6f9"}},{"code_sha256_prefix":"4612dca95f28aeaf","entry":"get_similarity_score","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/adverserial/adv_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/adverserial/adv_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4612dca95f28aeaf"}},{"code_sha256_prefix":"796b7eaaef2c4dd5","entry":"gpt_eval","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/disparagement/disp_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/disparagement/disp_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"796b7eaaef2c4dd5"}},{"code_sha256_prefix":"ed0ccb3bae64b137","entry":"process_csv_file","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/fairness-preference/fair_pref_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/fairness-preference/fair_pref_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed0ccb3bae64b137"}},{"code_sha256_prefix":"d2a53c7af3589e7d","entry":"remove_descent_variants","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/fairness-preference/fair_pref_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/fairness-preference/fair_pref_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2a53c7af3589e7d"}},{"code_sha256_prefix":"f99ee61006d6c28d","entry":"replacements","repo":"AikyamLab/clinic","repo_kind":"found_in_text","path":"evaluation/fairness-stereotype/fair_stereo_eval.py","file_url":"https://github.com/AikyamLab/clinic/blob/HEAD/evaluation/fairness-stereotype/fair_stereo_eval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f99ee61006d6c28d"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_api"},"syntology_extracted_results":null}