{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/ehrshot-an-ehr-benchmark-for-few-shot-1","title":"EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models","arxiv_id":"2307.02028","date":"2023-07-05","proceeding":"NeurIPS 2023 11","authors":["Michael Wornow","Rahul Thapa","Ethan Steinberg","Jason A. Fries","Nigam H. Shah"],"abstract":"While the general machine learning (ML) community has benefited from public datasets, tasks, and models, the progress of ML in healthcare has been hampered by a lack of such shared assets. The success of foundation models creates new challenges for healthcare ML by requiring access to shared pretrained models to validate performance benefits. We help address these challenges through three contributions. First, we publish a new dataset, EHRSHOT, which contains deidentified structured data from the electronic health records (EHRs) of 6,739 patients from Stanford Medicine. Unlike MIMIC-III/IV and other popular EHR datasets, EHRSHOT is longitudinal and not restricted to ICU/ED patients. Second, we publish the weights of CLMBR-T-base, a 141M parameter clinical foundation model pretrained on the structured EHR data of 2.57M patients. We are one of the first to fully release such a model for coded EHR data; in contrast, most prior models released for clinical data (e.g. GatorTron, ClinicalBERT) only work with unstructured text and cannot process the rich, structured data within an EHR. We provide an end-to-end pipeline for the community to validate and build upon its performance. Third, we define 15 few-shot clinical prediction tasks, enabling evaluation of foundation models on benefits such as sample efficiency and task adaptation. Our model and dataset are available via a research data use agreement from our website: https://ehrshot.stanford.edu. Code to reproduce our results are available at our Github repo: https://github.com/som-shahlab/ehrshot-benchmark","url_abs":"https://arxiv.org/abs/2307.02028v3","url_pdf":"https://arxiv.org/pdf/2307.02028v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"ehrshot-an-ehr-benchmark-for-few-shot-1","repo_url":"https://github.com/som-shahlab/ehrshot-benchmark","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"ehrshot-an-ehr-benchmark-for-few-shot-1","repo_url":"https://github.com/stefanhgm/ehrshot-benchmark","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2307.02028","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.02028"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/stefanhgm/ehrshot-benchmark","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/som-shahlab/ehrshot-benchmark","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":5,"unverified":11},"by_repo_kind":{"official":{"samples":9,"ran":5,"repositories":1},"listed":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e387d3515af3e462","entry":"calc_metrics","repo":"som-shahlab/ehrshot-benchmark","repo_kind":"official","path":"ehrshot/7b_eval_zero_shot.py","file_url":"https://github.com/som-shahlab/ehrshot-benchmark/blob/HEAD/ehrshot/7b_eval_zero_shot.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e387d3515af3e462"}},{"code_sha256_prefix":"26d24fff8949b09f","entry":"compute_random_num","repo":"som-shahlab/ehrshot-benchmark","repo_kind":"official","path":"ehrshot/labelers/core.py","file_url":"https://github.com/som-shahlab/ehrshot-benchmark/blob/HEAD/ehrshot/labelers/core.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"26d24fff8949b09f"}},{"code_sha256_prefix":"ff1a98172ef1054f","entry":"get_splits","repo":"som-shahlab/ehrshot-benchmark","repo_kind":"official","path":"ehrshot/9_make_cohort_plots.py","file_url":"https://github.com/som-shahlab/ehrshot-benchmark/blob/HEAD/ehrshot/9_make_cohort_plots.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ff1a98172ef1054f"}},{"code_sha256_prefix":"3111cc4393ec453b","entry":"lr_lambda","repo":"som-shahlab/ehrshot-benchmark","repo_kind":"official","path":"ehrshot/7_eval_finetune.py","file_url":"https://github.com/som-shahlab/ehrshot-benchmark/blob/HEAD/ehrshot/7_eval_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3111cc4393ec453b"}},{"code_sha256_prefix":"1ccac98a15f1c7bc","entry":"tune_hyperparams","repo":"som-shahlab/ehrshot-benchmark","repo_kind":"official","path":"ehrshot/7_eval_finetune.py","file_url":"https://github.com/som-shahlab/ehrshot-benchmark/blob/HEAD/ehrshot/7_eval_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1ccac98a15f1c7bc"}},{"code_sha256_prefix":"3eeebfadd62f674c","entry":"export_tuning_summary_table","repo":"stefanhgm/ehrshot-benchmark","repo_kind":"listed","path":"ehrshot/12_merge_and_plot_revision_results.py","file_url":"https://github.com/stefanhgm/ehrshot-benchmark/blob/HEAD/ehrshot/12_merge_and_plot_revision_results.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3eeebfadd62f674c"}},{"code_sha256_prefix":"937f2be12f9f3025","entry":"fit_logreg_lbfgs","repo":"som-shahlab/ehrshot-benchmark","repo_kind":"official","path":"ehrshot/7_eval_finetune.py","file_url":"https://github.com/som-shahlab/ehrshot-benchmark/blob/HEAD/ehrshot/7_eval_finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"937f2be12f9f3025"}},{"code_sha256_prefix":"b570adb57127d95c","entry":"format_icl_examples","repo":"stefanhgm/ehrshot-benchmark","repo_kind":"listed","path":"ehrshot/10b_fit_and_eval_decoder.py","file_url":"https://github.com/stefanhgm/ehrshot-benchmark/blob/HEAD/ehrshot/10b_fit_and_eval_decoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b570adb57127d95c"}},{"code_sha256_prefix":"f2dd7454f7178ecb","entry":"generate_shots","repo":"som-shahlab/ehrshot-benchmark","repo_kind":"official","path":"ehrshot/6_generate_shots.py","file_url":"https://github.com/som-shahlab/ehrshot-benchmark/blob/HEAD/ehrshot/6_generate_shots.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f2dd7454f7178ecb"}},{"code_sha256_prefix":"f591a2afaa8691c1","entry":"get_k_samples","repo":"som-shahlab/ehrshot-benchmark","repo_kind":"official","path":"ehrshot/6_generate_shots.py","file_url":"https://github.com/som-shahlab/ehrshot-benchmark/blob/HEAD/ehrshot/6_generate_shots.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f591a2afaa8691c1"}},{"code_sha256_prefix":"0dec51f61a914fe7","entry":"guo_los_zeroshot_rollouts","repo":"som-shahlab/ehrshot-benchmark","repo_kind":"official","path":"ehrshot/7b_eval_zero_shot.py","file_url":"https://github.com/som-shahlab/ehrshot-benchmark/blob/HEAD/ehrshot/7b_eval_zero_shot.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0dec51f61a914fe7"}},{"code_sha256_prefix":"da830e412673ca56","entry":"import_run_evaluation","repo":"stefanhgm/ehrshot-benchmark","repo_kind":"listed","path":"ehrshot/20_run_llm_variants_experiments.py","file_url":"https://github.com/stefanhgm/ehrshot-benchmark/blob/HEAD/ehrshot/20_run_llm_variants_experiments.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"da830e412673ca56"}},{"code_sha256_prefix":"2612d55d41fef50d","entry":"load_data","repo":"stefanhgm/ehrshot-benchmark","repo_kind":"listed","path":"ehrshot/10a_fit_and_eval_encoder.py","file_url":"https://github.com/stefanhgm/ehrshot-benchmark/blob/HEAD/ehrshot/10a_fit_and_eval_encoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2612d55d41fef50d"}},{"code_sha256_prefix":"61478e91a0817d84","entry":"resolve_instruction_from_serializations","repo":"stefanhgm/ehrshot-benchmark","repo_kind":"listed","path":"ehrshot/10a_fit_and_eval_encoder.py","file_url":"https://github.com/stefanhgm/ehrshot-benchmark/blob/HEAD/ehrshot/10a_fit_and_eval_encoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"61478e91a0817d84"}},{"code_sha256_prefix":"29373a416d078cca","entry":"select_icl_example_indices","repo":"stefanhgm/ehrshot-benchmark","repo_kind":"listed","path":"ehrshot/10b_fit_and_eval_decoder.py","file_url":"https://github.com/stefanhgm/ehrshot-benchmark/blob/HEAD/ehrshot/10b_fit_and_eval_decoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"29373a416d078cca"}},{"code_sha256_prefix":"b0a8dc17cfae1347","entry":"validate_icl_shots","repo":"stefanhgm/ehrshot-benchmark","repo_kind":"listed","path":"ehrshot/10b_fit_and_eval_decoder.py","file_url":"https://github.com/stefanhgm/ehrshot-benchmark/blob/HEAD/ehrshot/10b_fit_and_eval_decoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b0a8dc17cfae1347"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}