{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/yet-another-icu-benchmark-a-flexible-multi","title":"Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML","arxiv_id":"2306.05109","date":"2023-06-08","proceeding":null,"authors":["Robin van de Water","Hendrik Schmidt","Paul Elbers","Patrick Thoral","Bert Arnrich","Patrick Rockenschaub"],"abstract":"Medical applications of machine learning (ML) have experienced a surge in popularity in recent years. The intensive care unit (ICU) is a natural habitat for ML given the abundance of available data from electronic health records. Models have been proposed to address numerous ICU prediction tasks like the early detection of complications. While authors frequently report state-of-the-art performance, it is challenging to verify claims of superiority. Datasets and code are not always published, and cohort definitions, preprocessing pipelines, and training setups are difficult to reproduce. This work introduces Yet Another ICU Benchmark (YAIB), a modular framework that allows researchers to define reproducible and comparable clinical ML experiments; we offer an end-to-end solution from cohort definition to model evaluation. The framework natively supports most open-access ICU datasets (MIMIC III/IV, eICU, HiRID, AUMCdb) and is easily adaptable to future ICU datasets. Combined with a transparent preprocessing pipeline and extensible training code for multiple ML and deep learning models, YAIB enables unified model development. Our benchmark comes with five predefined established prediction tasks (mortality, acute kidney injury, sepsis, kidney function, and length of stay) developed in collaboration with clinicians. Adding further tasks is straightforward by design. Using YAIB, we demonstrate that the choice of dataset, cohort definition, and preprocessing have a major impact on the prediction performance - often more so than model class - indicating an urgent need for YAIB as a holistic benchmarking tool. We provide our work to the clinical ML community to accelerate method development and enable real-world clinical implementations. Software Repository: https://github.com/rvandewater/YAIB.","url_abs":"https://arxiv.org/abs/2306.05109v4","url_pdf":"https://arxiv.org/pdf/2306.05109v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"yet-another-icu-benchmark-a-flexible-multi","repo_url":"https://github.com/rvandewater/recipys","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"yet-another-icu-benchmark-a-flexible-multi","repo_url":"https://github.com/rvandewater/yaib","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"yet-another-icu-benchmark-a-flexible-multi","repo_url":"https://github.com/rvandewater/yaib-cohorts","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"yet-another-icu-benchmark-a-flexible-multi","repo_url":"https://github.com/rvandewater/yaib-models","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"},{"task_slug":"kidney-function","task_name":"Kidney Function"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2306.05109","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.05109"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rvandewater/yaib","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rvandewater/recipys","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rvandewater/yaib-models","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rvandewater/yaib-cohorts","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":3,"unverified":22},"by_repo_kind":{"official":{"samples":25,"ran":3,"repositories":3}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0f786c407fb1ee4c","entry":"swish","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/imputation/diffwave.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/imputation/diffwave.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":2,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0f786c407fb1ee4c"}},{"code_sha256_prefix":"7340b482ffe44836","entry":"Conv1d_with_init","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/imputation/csdi.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/imputation/csdi.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7340b482ffe44836"}},{"code_sha256_prefix":"e706793d7d22ae44","entry":"get_torch_trans","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/imputation/csdi.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/imputation/csdi.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e706793d7d22ae44"}},{"code_sha256_prefix":"6e037d50ada32ff6","entry":"BO_mask","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/imputation/amputations.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/imputation/amputations.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6e037d50ada32ff6"}},{"code_sha256_prefix":"23a54dd969a6cc4e","entry":"MAR_logistic_mask","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/imputation/amputations.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/imputation/amputations.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"23a54dd969a6cc4e"}},{"code_sha256_prefix":"c9b8608abd0c8fda","entry":"MCAR_mask","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/imputation/amputations.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/imputation/amputations.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c9b8608abd0c8fda"}},{"code_sha256_prefix":"913d82af53065418","entry":"accuracy","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/models/custom_metrics.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/models/custom_metrics.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"913d82af53065418"}},{"code_sha256_prefix":"0741c6a2d1c22012","entry":"apply_wandb_sweep","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/wandb_utils.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/wandb_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0741c6a2d1c22012"}},{"code_sha256_prefix":"38373950dd739e60","entry":"calc_diffusion_hyperparams","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/imputation/diffwave.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/imputation/diffwave.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38373950dd739e60"}},{"code_sha256_prefix":"5e384ebd2c7aa5ca","entry":"calc_diffusion_step_embedding","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/imputation/diffwave.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/imputation/diffwave.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e384ebd2c7aa5ca"}},{"code_sha256_prefix":"92336a4643e58418","entry":"confusion_matrix","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/models/custom_metrics.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/models/custom_metrics.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"92336a4643e58418"}},{"code_sha256_prefix":"24a54df4893018f7","entry":"create_optimizer","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/models/utils.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/models/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"24a54df4893018f7"}},{"code_sha256_prefix":"a5e1412c8a01d50a","entry":"create_run_dir","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/run_utils.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/run_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a5e1412c8a01d50a"}},{"code_sha256_prefix":"270211d5760b3ffc","entry":"create_scheduler","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/models/utils.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/models/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"270211d5760b3ffc"}},{"code_sha256_prefix":"8e82af6cb2d87a0a","entry":"enlist_dt","repo":"rvandewater/recipys","repo_kind":"official","path":"src/recipies/selector.py","file_url":"https://github.com/rvandewater/recipys/blob/HEAD/src/recipies/selector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e82af6cb2d87a0a"}},{"code_sha256_prefix":"0dbb1b92243bed71","entry":"enlist_str","repo":"rvandewater/recipys","repo_kind":"official","path":"src/recipies/selector.py","file_url":"https://github.com/rvandewater/recipys/blob/HEAD/src/recipies/selector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0dbb1b92243bed71"}},{"code_sha256_prefix":"2482794f8f0230bc","entry":"generate_generic_datetime_data","repo":"rvandewater/recipys","repo_kind":"official","path":"benchmarking/generate_data.py","file_url":"https://github.com/rvandewater/recipys/blob/HEAD/benchmarking/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2482794f8f0230bc"}},{"code_sha256_prefix":"4a029ddd637df2c4","entry":"generate_icu_data","repo":"rvandewater/recipys","repo_kind":"official","path":"benchmarking/generate_data.py","file_url":"https://github.com/rvandewater/recipys/blob/HEAD/benchmarking/generate_data.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a029ddd637df2c4"}},{"code_sha256_prefix":"20a234617c6040ca","entry":"get_config_files","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/run_utils.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/run_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20a234617c6040ca"}},{"code_sha256_prefix":"aaf9a507a8257e69","entry":"intersection","repo":"rvandewater/recipys","repo_kind":"official","path":"src/recipies/selector.py","file_url":"https://github.com/rvandewater/recipys/blob/HEAD/src/recipies/selector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aaf9a507a8257e69"}},{"code_sha256_prefix":"c3894948f22f015a","entry":"load_pretrained_imputation_model","repo":"rvandewater/yaib","repo_kind":"official","path":"icu_benchmarks/run_utils.py","file_url":"https://github.com/rvandewater/yaib/blob/HEAD/icu_benchmarks/run_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c3894948f22f015a"}},{"code_sha256_prefix":"3807bbd470276d2b","entry":"make_grid_mapper","repo":"rvandewater/yaib-cohorts","repo_kind":"official","path":"Python/src/ricu_utils.py","file_url":"https://github.com/rvandewater/yaib-cohorts/blob/HEAD/Python/src/ricu_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3807bbd470276d2b"}},{"code_sha256_prefix":"3ba0466f618bc25e","entry":"make_patient_mapper","repo":"rvandewater/yaib-cohorts","repo_kind":"official","path":"Python/src/ricu_utils.py","file_url":"https://github.com/rvandewater/yaib-cohorts/blob/HEAD/Python/src/ricu_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3ba0466f618bc25e"}},{"code_sha256_prefix":"fe67bfaddfb0cd3f","entry":"make_train_test","repo":"rvandewater/yaib-cohorts","repo_kind":"official","path":"Python/utils.py","file_url":"https://github.com/rvandewater/yaib-cohorts/blob/HEAD/Python/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe67bfaddfb0cd3f"}},{"code_sha256_prefix":"c176141a276face1","entry":"stop_window_at","repo":"rvandewater/yaib-cohorts","repo_kind":"official","path":"Python/src/ricu_utils.py","file_url":"https://github.com/rvandewater/yaib-cohorts/blob/HEAD/Python/src/ricu_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c176141a276face1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}