{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/meditron-70b-scaling-medical-pretraining-for","title":"MEDITRON-70B: Scaling Medical Pretraining for Large Language Models","arxiv_id":"2311.16079","date":"2023-11-27","proceeding":null,"authors":["Zeming Chen","Alejandro Hernández Cano","Angelika Romanou","Antoine Bonnet","Kyle Matoba","Francesco Salvi","Matteo Pagliardini","Simin Fan","Andreas Köpf","Amirkeivan Mohtashami","Alexandre Sallinen","Alireza Sakhaeirad","Vinitra Swamy","Igor Krawczuk","Deniz Bayazit","Axel Marmet","Syrielle Montariol","Mary-Anne Hartley","Martin Jaggi","Antoine Bosselut"],"abstract":"Large language models (LLMs) can potentially democratize access to medical knowledge. While many efforts have been made to harness and improve LLMs' medical knowledge and reasoning capacities, the resulting models are either closed-source (e.g., PaLM, GPT-4) or limited in scale (<= 13B parameters), which restricts their abilities. In this work, we improve access to large-scale medical LLMs by releasing MEDITRON: a suite of open-source LLMs with 7B and 70B parameters adapted to the medical domain. MEDITRON builds on Llama-2 (through our adaptation of Nvidia's Megatron-LM distributed trainer), and extends pretraining on a comprehensively curated medical corpus, including selected PubMed articles, abstracts, and internationally-recognized medical guidelines. Evaluations using four major medical benchmarks show significant performance gains over several state-of-the-art baselines before and after task-specific finetuning. Overall, MEDITRON achieves a 6% absolute performance gain over the best public baseline in its parameter class and 3% over the strongest baseline we finetuned from Llama-2. Compared to closed-source LLMs, MEDITRON-70B outperforms GPT-3.5 and Med-PaLM and is within 5% of GPT-4 and 10% of Med-PaLM-2. We release our code for curating the medical pretraining corpus and the MEDITRON model weights to drive open-source development of more capable medical LLMs.","url_abs":"https://arxiv.org/abs/2311.16079v1","url_pdf":"https://arxiv.org/pdf/2311.16079v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"meditron-70b-scaling-medical-pretraining-for","repo_url":"https://github.com/epfllm/meditron","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"articles","task_name":"Articles"},{"task_slug":"conditional-text-generation","task_name":"Conditional Text Generation"},{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"multiple-choice-qa","task_name":"Multiple Choice Question Answering (MCQA)"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-4","method_name":"GPT-4"},{"method_slug":"llama","method_name":"LLaMA"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"palm","method_name":"PaLM"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/few-shot-learning-on-medconceptsqa","task":"Few-Shot Learning","dataset":"MedConceptsQA","model":"epfl-llm/meditron-70b","rank_in_archive_order":7,"of":12,"metrics":{"Accuracy":"25.262"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-learning-on-medconceptsqa","task":"Few-Shot Learning","dataset":"MedConceptsQA","model":"epfl-llm/meditron-7b","rank_in_archive_order":12,"of":12,"metrics":{"Accuracy":"23.787"},"uses_additional_data":false},{"leaderboard":"/sota/multiple-choice-question-answering-mcqa-on-21","task":"Multiple Choice Question Answering (MCQA)","dataset":"MedMCQA","model":"Meditron-70B (CoT + SC)","rank_in_archive_order":11,"of":22,"metrics":{"Dev Set (Acc-%)":"66.0"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-medqa-usmle","task":"Question Answering","dataset":"MedQA","model":"Meditron-70B (CoT + SC)","rank_in_archive_order":9,"of":27,"metrics":{"Accuracy":"70.2"},"uses_additional_data":false},{"leaderboard":"/sota/question-answering-on-medqa-usmle","task":"Question Answering","dataset":"MedQA","model":"LLAMA-2 (70B SC CoT)","rank_in_archive_order":11,"of":27,"metrics":{"Accuracy":"61.5"},"uses_additional_data":true},{"leaderboard":"/sota/question-answering-on-medqa-usmle","task":"Question Answering","dataset":"MedQA","model":"LLAMA-2 (70B)","rank_in_archive_order":14,"of":27,"metrics":{"Accuracy":"59.2"},"uses_additional_data":true},{"leaderboard":"/sota/question-answering-on-pubmedqa","task":"Question Answering","dataset":"PubMedQA","model":"Meditron-70B (CoT + SC)","rank_in_archive_order":1,"of":30,"metrics":{"Accuracy":"81.6"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-learning-on-medconceptsqa","task":"Zero-Shot Learning","dataset":"MedConceptsQA","model":"epfl-llm/meditron-7b","rank_in_archive_order":5,"of":13,"metrics":{"Accuracy":"25.751"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-learning-on-medconceptsqa","task":"Zero-Shot Learning","dataset":"MedConceptsQA","model":"epfl-llm/meditron-70b","rank_in_archive_order":8,"of":13,"metrics":{"Accuracy":"25.360"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.16079","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.16079"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/epfllm/meditron","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":9,"unverified":5},"by_repo_kind":{"official":{"samples":14,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4452a22b84309806","entry":"clean_mcq_answer","repo":"epfllm/meditron","repo_kind":"official","path":"evaluation/evaluate.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4452a22b84309806"}},{"code_sha256_prefix":"aeb12ea36b675148","entry":"concatenate_sections","repo":"epfllm/meditron","repo_kind":"official","path":"gap-replay/guidelines/clean.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/gap-replay/guidelines/clean.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"aeb12ea36b675148"}},{"code_sha256_prefix":"983fd1e0e6e34a3f","entry":"format_mcq","repo":"epfllm/meditron","repo_kind":"official","path":"evaluation/benchmarks.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/evaluation/benchmarks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"983fd1e0e6e34a3f"}},{"code_sha256_prefix":"1ababc7e1e681f41","entry":"format_prompt","repo":"epfllm/meditron","repo_kind":"official","path":"evaluation/inference.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/evaluation/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ababc7e1e681f41"}},{"code_sha256_prefix":"15943fc5c02ac67e","entry":"get_mesh_tags","repo":"epfllm/meditron","repo_kind":"official","path":"gap-replay/pubmed/augment.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/gap-replay/pubmed/augment.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"15943fc5c02ac67e"}},{"code_sha256_prefix":"8143fb4298119175","entry":"infer_ndocs","repo":"epfllm/meditron","repo_kind":"official","path":"finetuning/sft.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/finetuning/sft.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8143fb4298119175"}},{"code_sha256_prefix":"7e8c378eedc74237","entry":"load_json","repo":"epfllm/meditron","repo_kind":"official","path":"evaluation/evaluate.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e8c378eedc74237"}},{"code_sha256_prefix":"ed2919d070e3e6e8","entry":"load_jsonl","repo":"epfllm/meditron","repo_kind":"official","path":"evaluation/evaluate.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/evaluation/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ed2919d070e3e6e8"}},{"code_sha256_prefix":"63ecbb0137ec2cf0","entry":"read_jsonl","repo":"epfllm/meditron","repo_kind":"official","path":"gap-replay/guidelines/clean.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/gap-replay/guidelines/clean.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63ecbb0137ec2cf0"}},{"code_sha256_prefix":"df54cf786ee59bad","entry":"benchmark_factory","repo":"epfllm/meditron","repo_kind":"official","path":"evaluation/benchmarks.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/evaluation/benchmarks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df54cf786ee59bad"}},{"code_sha256_prefix":"baf77bfa09d1505d","entry":"load_instruction","repo":"epfllm/meditron","repo_kind":"official","path":"evaluation/benchmarks.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/evaluation/benchmarks.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"baf77bfa09d1505d"}},{"code_sha256_prefix":"42f0d9df1f4e7733","entry":"tokenize_data","repo":"epfllm/meditron","repo_kind":"official","path":"finetuning/sft.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/finetuning/sft.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"42f0d9df1f4e7733"}},{"code_sha256_prefix":"bc10fd06d25d060b","entry":"tokenizer_param","repo":"epfllm/meditron","repo_kind":"official","path":"evaluation/inference.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/evaluation/inference.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bc10fd06d25d060b"}},{"code_sha256_prefix":"6c12a5dfd0746a7c","entry":"update_metadata_mesh","repo":"epfllm/meditron","repo_kind":"official","path":"gap-replay/pubmed/augment.py","file_url":"https://github.com/epfllm/meditron/blob/HEAD/gap-replay/pubmed/augment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6c12a5dfd0746a7c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}