{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/icym2i-the-illusion-of-multimodal","title":"ICYM2I: The illusion of multimodal informativeness under missingness","arxiv_id":"2505.16953","date":"2025-05-22","proceeding":null,"authors":["Young Sang Choi","Vincent Jeanselme","Pierre Elias","Shalmali Joshi"],"abstract":"Multimodal learning is of continued interest in artificial intelligence-based applications, motivated by the potential information gain from combining different types of data. However, modalities collected and curated during development may differ from the modalities available at deployment due to multiple factors including cost, hardware failure, or -- as we argue in this work -- the perceived informativeness of a given modality. Na{\\\"i}ve estimation of the information gain associated with including an additional modality without accounting for missingness may result in improper estimates of that modality's value in downstream tasks. Our work formalizes the problem of missingness in multimodal learning and demonstrates the biases resulting from ignoring this process. To address this issue, we introduce ICYM2I (In Case You Multimodal Missed It), a framework for the evaluation of predictive performance and information gain under missingness through inverse probability weighting-based correction. We demonstrate the importance of the proposed adjustment to estimate information gain under missingness on synthetic, semi-synthetic, and real-world medical datasets.","url_abs":"https://arxiv.org/abs/2505.16953v1","url_pdf":"https://arxiv.org/pdf/2505.16953v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"icym2i-the-illusion-of-multimodal","repo_url":"https://github.com/reAIM-Lab/ICYM2I","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"icym2i-the-illusion-of-multimodal","repo_url":"https://github.com/reAIM-Lab/ehr_foundation_model_benchmark","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"informativeness","task_name":"Informativeness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2505.16953","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.16953"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/reAIM-Lab/ehr_foundation_model_benchmark","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/reAIM-Lab/ICYM2I","reach":null}],"summary":{"ran":1,"ran_draft_wrong":4,"unverified":1},"by_repo_kind":{"official":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"119e8ba571101a6d","entry":"entropy_from_joint","repo":"reAIM-Lab/ICYM2I","repo_kind":"official","path":"information_decomposition/multi_pid.py","file_url":"https://github.com/reAIM-Lab/ICYM2I/blob/HEAD/information_decomposition/multi_pid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"119e8ba571101a6d"}},{"code_sha256_prefix":"51856d55f063f8d4","entry":"three_way_mutual_information","repo":"reAIM-Lab/ICYM2I","repo_kind":"official","path":"information_decomposition/multi_pid.py","file_url":"https://github.com/reAIM-Lab/ICYM2I/blob/HEAD/information_decomposition/multi_pid.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51856d55f063f8d4"}},{"code_sha256_prefix":"7299bb157ef2849e","entry":"three_way_mutual_information_entropy","repo":"reAIM-Lab/ICYM2I","repo_kind":"official","path":"information_decomposition/multi_pid.py","file_url":"https://github.com/reAIM-Lab/ICYM2I/blob/HEAD/information_decomposition/multi_pid.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7299bb157ef2849e"}},{"code_sha256_prefix":"dbdc4a6f805e2261","entry":"two_way_mutual_information","repo":"reAIM-Lab/ICYM2I","repo_kind":"official","path":"information_decomposition/multi_pid.py","file_url":"https://github.com/reAIM-Lab/ICYM2I/blob/HEAD/information_decomposition/multi_pid.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dbdc4a6f805e2261"}},{"code_sha256_prefix":"2c65735b65ef2a21","entry":"two_way_mutual_information_entropy","repo":"reAIM-Lab/ICYM2I","repo_kind":"official","path":"information_decomposition/multi_pid.py","file_url":"https://github.com/reAIM-Lab/ICYM2I/blob/HEAD/information_decomposition/multi_pid.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2c65735b65ef2a21"}},{"code_sha256_prefix":"a7dcf36fbbc320c7","entry":"pid_decomposition","repo":"reAIM-Lab/ICYM2I","repo_kind":"official","path":"information_decomposition/multi_pid.py","file_url":"https://github.com/reAIM-Lab/ICYM2I/blob/HEAD/information_decomposition/multi_pid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a7dcf36fbbc320c7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}