{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/medeval-a-multi-level-multi-task-and-multi","title":"MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation","arxiv_id":"2310.14088","date":"2023-10-21","proceeding":null,"authors":["Zexue He","Yu Wang","An Yan","Yao Liu","Eric Y. Chang","Amilcare Gentili","Julian McAuley","Chun-Nan Hsu"],"abstract":"Curated datasets for healthcare are often limited due to the need of human annotations from experts. In this paper, we present MedEval, a multi-level, multi-task, and multi-domain medical benchmark to facilitate the development of language models for healthcare. MedEval is comprehensive and consists of data from several healthcare systems and spans 35 human body regions from 8 examination modalities. With 22,779 collected sentences and 21,228 reports, we provide expert annotations at multiple levels, offering a granular potential usage of the data and supporting a wide range of tasks. Moreover, we systematically evaluated 10 generic and domain-specific language models under zero-shot and finetuning settings, from domain-adapted baselines in healthcare to general-purposed state-of-the-art large language models (e.g., ChatGPT). Our evaluations reveal varying effectiveness of the two categories of language models across different tasks, from which we notice the importance of instruction tuning for few-shot usage of large language models. Our investigation paves the way toward benchmarking language models for healthcare and provides valuable insights into the strengths and limitations of adopting large language models in medical domains, informing their practical applications and future advancements.","url_abs":"https://arxiv.org/abs/2310.14088v3","url_pdf":"https://arxiv.org/pdf/2310.14088v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"language-model-evaluation","task_name":"Language Model Evaluation"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.14088","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.14088"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/ZexueHe/MedEval","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/chapmanbe/pyConTextNLP","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"dbe566142582f29e","entry":"get_document_markups","repo":"chapmanbe/pyConTextNLP","repo_kind":"found_in_text","path":"pyConTextNLP/utils.py","file_url":"https://github.com/chapmanbe/pyConTextNLP/blob/HEAD/pyConTextNLP/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dbe566142582f29e"}},{"code_sha256_prefix":"cd5db11c56a68c86","entry":"get_section_markups","repo":"chapmanbe/pyConTextNLP","repo_kind":"found_in_text","path":"pyConTextNLP/utils.py","file_url":"https://github.com/chapmanbe/pyConTextNLP/blob/HEAD/pyConTextNLP/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cd5db11c56a68c86"}},{"code_sha256_prefix":"0def503313cb8bca","entry":"mark_text","repo":"chapmanbe/pyConTextNLP","repo_kind":"found_in_text","path":"pyConTextNLP/display/html.py","file_url":"https://github.com/chapmanbe/pyConTextNLP/blob/HEAD/pyConTextNLP/display/html.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0def503313cb8bca"}},{"code_sha256_prefix":"07e6e332127e12fe","entry":"xmlScrub","repo":"chapmanbe/pyConTextNLP","repo_kind":"found_in_text","path":"pyConTextNLP/io/xml.py","file_url":"https://github.com/chapmanbe/pyConTextNLP/blob/HEAD/pyConTextNLP/io/xml.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07e6e332127e12fe"}},{"code_sha256_prefix":"b159fbd532242173","entry":"get_items","repo":"chapmanbe/pyConTextNLP","repo_kind":"found_in_text","path":"pyConTextNLP/itemData.py","file_url":"https://github.com/chapmanbe/pyConTextNLP/blob/HEAD/pyConTextNLP/itemData.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b159fbd532242173"}},{"code_sha256_prefix":"4bb3688472aef7a8","entry":"mark_document_with_html","repo":"chapmanbe/pyConTextNLP","repo_kind":"found_in_text","path":"pyConTextNLP/display/html.py","file_url":"https://github.com/chapmanbe/pyConTextNLP/blob/HEAD/pyConTextNLP/display/html.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bb3688472aef7a8"}},{"code_sha256_prefix":"a0dccff7102fadfb","entry":"mark_document_with_html_sections","repo":"chapmanbe/pyConTextNLP","repo_kind":"found_in_text","path":"pyConTextNLP/display/html.py","file_url":"https://github.com/chapmanbe/pyConTextNLP/blob/HEAD/pyConTextNLP/display/html.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a0dccff7102fadfb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}