{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/integrated-multimodal-artificial-intelligence","title":"Integrated multimodal artificial intelligence framework for healthcare applications","arxiv_id":"2202.12998","date":"2022-02-25","proceeding":null,"authors":["Luis R. Soenksen","Yu Ma","Cynthia Zeng","Leonard D. J. Boussioux","Kimberly Villalobos Carballo","Liangyuan Na","Holly M. Wiberg","Michael L. Li","Ignacio Fuentes","Dimitris Bertsimas"],"abstract":"Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We evaluate our HAIM framework by training and characterizing 14,324 independent models based on HAIM-MIMIC-MM, a multimodal clinical database (N=34,537 samples) containing 7,279 unique hospitalizations and 6,485 patients, spanning all possible input combinations of 4 data modalities (i.e., tabular, time-series, text, and images), 11 unique data sources and 12 predictive tasks. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations (by 6-33%), including 10 distinct chest pathology diagnoses, along with length-of-stay and 48-hour mortality predictions. We also quantify the contribution of each modality and data source using Shapley values, which demonstrates the heterogeneity in data modality importance and the necessity of multimodal inputs across different healthcare-relevant tasks. The generalizable properties and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a promising pathway for future multimodal predictive systems in clinical and operational healthcare settings.","url_abs":"https://arxiv.org/abs/2202.12998v4","url_pdf":"https://arxiv.org/pdf/2202.12998v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"integrated-multimodal-artificial-intelligence","repo_url":"https://github.com/lrsoenksen/haim","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"time-series","task_name":"Time Series Analysis"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2202.12998","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2202.12998"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lrsoenksen/haim","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":6},"by_repo_kind":{"official":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9ebe0ca5a1bbda84","entry":"date_diff_hrs","repo":"lrsoenksen/haim","repo_kind":"official","path":"MIMIC_IV_HAIM_API.py","file_url":"https://github.com/lrsoenksen/haim/blob/HEAD/MIMIC_IV_HAIM_API.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9ebe0ca5a1bbda84"}},{"code_sha256_prefix":"22923f9091b05b3a","entry":"get_patient_icustay","repo":"lrsoenksen/haim","repo_kind":"official","path":"MIMIC_IV_HAIM_API.py","file_url":"https://github.com/lrsoenksen/haim/blob/HEAD/MIMIC_IV_HAIM_API.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"22923f9091b05b3a"}},{"code_sha256_prefix":"f0a6e2b2c5d3d8f7","entry":"get_scores_clf","repo":"lrsoenksen/haim","repo_kind":"official","path":"prediction_util.py","file_url":"https://github.com/lrsoenksen/haim/blob/HEAD/prediction_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f0a6e2b2c5d3d8f7"}},{"code_sha256_prefix":"a154c9a7ebab4c9b","entry":"get_timebound_patient_icustay","repo":"lrsoenksen/haim","repo_kind":"official","path":"MIMIC_IV_HAIM_API.py","file_url":"https://github.com/lrsoenksen/haim/blob/HEAD/MIMIC_IV_HAIM_API.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a154c9a7ebab4c9b"}},{"code_sha256_prefix":"b724aa44fd0c9636","entry":"run_xgb","repo":"lrsoenksen/haim","repo_kind":"official","path":"prediction_util.py","file_url":"https://github.com/lrsoenksen/haim/blob/HEAD/prediction_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b724aa44fd0c9636"}},{"code_sha256_prefix":"d983f341a80db5a1","entry":"update_result","repo":"lrsoenksen/haim","repo_kind":"official","path":"prediction_util.py","file_url":"https://github.com/lrsoenksen/haim/blob/HEAD/prediction_util.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d983f341a80db5a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}