{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-evaluating-and-building-versatile","title":"Towards Evaluating and Building Versatile Large Language Models for Medicine","arxiv_id":"2408.12547","date":"2024-08-22","proceeding":null,"authors":["Chaoyi Wu","Pengcheng Qiu","Jinxin Liu","Hongfei Gu","Na Li","Ya zhang","Yanfeng Wang","Weidi Xie"],"abstract":"In this study, we present MedS-Bench, a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. Unlike existing benchmarks that focus on multiple-choice question answering, MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation, among others. We evaluated six leading LLMs, e.g., MEDITRON, Mistral, InternLM 2, Llama 3, GPT-4, and Claude-3.5 using few-shot prompting, and found that even the most sophisticated models struggle with these complex tasks. To address these limitations, we developed MedS-Ins, a large-scale instruction tuning dataset for medicine. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks. To demonstrate the dataset's utility, we conducted a proof-of-concept experiment by performing instruction tuning on a lightweight, open-source medical language model. The resulting model, MMedIns-Llama 3, significantly outperformed existing models across nearly all clinical tasks. To promote further advancements in the application of LLMs to clinical challenges, we have made the MedS-Ins dataset fully accessible and invite the research community to contribute to its expansion.Additionally, we have launched a dynamic leaderboard for MedS-Bench, which we plan to regularly update the test set to track progress and enhance the adaptation of general LLMs to the medical domain. Leaderboard: https://henrychur.github.io/MedS-Bench/. Github: https://github.com/MAGIC-AI4Med/MedS-Ins.","url_abs":"https://arxiv.org/abs/2408.12547v2","url_pdf":"https://arxiv.org/pdf/2408.12547v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-evaluating-and-building-versatile","repo_url":"https://github.com/magic-ai4med/meds-ins","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"multiple-choice","task_name":"Multiple-choice"},{"task_slug":"named-entity-recognition-1","task_name":"Named Entity Recognition"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"named-entity-recognition","task_name":"named-entity-recognition"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"focus","method_name":"Focus"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2408.12547","atlas_url":"https://app.syntology.ai/?focus=2408.12547","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2408.12547"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/magic-ai4med/meds-ins","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":7},"by_repo_kind":{"official":{"samples":7,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"62422cfd04737f43","entry":"clinical_outcome_accuracy","repo":"magic-ai4med/meds-ins","repo_kind":"official","path":"Metrics/ClinicalOutcomePrediction.py","file_url":"https://github.com/magic-ai4med/meds-ins/blob/HEAD/Metrics/ClinicalOutcomePrediction.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"62422cfd04737f43"}},{"code_sha256_prefix":"3bfa0964e50a53f0","entry":"diagnosis_accuracy","repo":"magic-ai4med/meds-ins","repo_kind":"official","path":"Metrics/Diagnosis.py","file_url":"https://github.com/magic-ai4med/meds-ins/blob/HEAD/Metrics/Diagnosis.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3bfa0964e50a53f0"}},{"code_sha256_prefix":"e8bab91eef87ee9b","entry":"hard_ner_f1","repo":"magic-ai4med/meds-ins","repo_kind":"official","path":"Metrics/NamedEntityRecognition.py","file_url":"https://github.com/magic-ai4med/meds-ins/blob/HEAD/Metrics/NamedEntityRecognition.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e8bab91eef87ee9b"}},{"code_sha256_prefix":"5aeddcce1d090dea","entry":"information_extraction_accuracy","repo":"magic-ai4med/meds-ins","repo_kind":"official","path":"Metrics/InformationExtraction.py","file_url":"https://github.com/magic-ai4med/meds-ins/blob/HEAD/Metrics/InformationExtraction.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5aeddcce1d090dea"}},{"code_sha256_prefix":"903410b9a3d46797","entry":"mcqa_accuracy","repo":"magic-ai4med/meds-ins","repo_kind":"official","path":"Metrics/MCQA.py","file_url":"https://github.com/magic-ai4med/meds-ins/blob/HEAD/Metrics/MCQA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"903410b9a3d46797"}},{"code_sha256_prefix":"e43b2ff01d72a919","entry":"nli_accuracy","repo":"magic-ai4med/meds-ins","repo_kind":"official","path":"Metrics/NatureLanguageInference.py","file_url":"https://github.com/magic-ai4med/meds-ins/blob/HEAD/Metrics/NatureLanguageInference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e43b2ff01d72a919"}},{"code_sha256_prefix":"340961a1e2c4108d","entry":"parse_entities","repo":"magic-ai4med/meds-ins","repo_kind":"official","path":"Metrics/NamedEntityRecognition.py","file_url":"https://github.com/magic-ai4med/meds-ins/blob/HEAD/Metrics/NamedEntityRecognition.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"340961a1e2c4108d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}