{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/quantifying-modeling-multimodal-interactions-1","title":"Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework","arxiv_id":"2302.12247","date":"2023-02-23","proceeding":"NeurIPS 2023 11","authors":["Paul Pu Liang","Yun Cheng","Xiang Fan","Chun Kai Ling","Suzanne Nie","Richard Chen","Zihao Deng","Nicholas Allen","Randy Auerbach","Faisal Mahmood","Ruslan Salakhutdinov","Louis-Philippe Morency"],"abstract":"The recent explosion of interest in multimodal applications has resulted in a wide selection of datasets and methods for representing and integrating information from different modalities. Despite these empirical advances, there remain fundamental research questions: How can we quantify the interactions that are necessary to solve a multimodal task? Subsequently, what are the most suitable multimodal models to capture these interactions? To answer these questions, we propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. We term these three measures as the PID statistics of a multimodal distribution (or PID for short), and introduce two new estimators for these PID statistics that scale to high-dimensional distributions. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks where PID estimations are compared with human annotations. Finally, we demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies engaging with domain experts in pathology, mood prediction, and robotic perception where our framework helps to recommend strong multimodal models for each application.","url_abs":"https://arxiv.org/abs/2302.12247v5","url_pdf":"https://arxiv.org/pdf/2302.12247v5.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"quantifying-modeling-multimodal-interactions-1","repo_url":"https://github.com/pliang279/pid","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"model-selection","task_name":"Model Selection"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.12247","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.12247"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/pliang279/PID","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pliang279/pid","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":10,"ran_violates":1,"ran_draft_wrong":4,"unverified":1},"by_repo_kind":{"official":{"samples":16,"ran":15,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"da154993a013ef1d","entry":"CI","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"da154993a013ef1d"}},{"code_sha256_prefix":"ba0c1a2505d20c59","entry":"CoI","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba0c1a2505d20c59"}},{"code_sha256_prefix":"5dd04c930a0883b6","entry":"MI","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5dd04c930a0883b6"}},{"code_sha256_prefix":"3a6eae326a67768e","entry":"UI","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a6eae326a67768e"}},{"code_sha256_prefix":"32886135fe06536a","entry":"convert_data_to_distribution","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32886135fe06536a"}},{"code_sha256_prefix":"d30928c06a648d94","entry":"extract_categorical_from_data","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d30928c06a648d94"}},{"code_sha256_prefix":"6de22f9453a75297","entry":"get_bounds","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6de22f9453a75297"}},{"code_sha256_prefix":"3a7e38eff2e450b8","entry":"get_quantities","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3a7e38eff2e450b8"}},{"code_sha256_prefix":"31bbe15992d2ab17","entry":"majorization_bound","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"31bbe15992d2ab17"}},{"code_sha256_prefix":"235a0f630ddd0692","entry":"simple_discrim","repo":"pliang279/pid","repo_kind":"official","path":"estimators/ce_alignment_information.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/estimators/ce_alignment_information.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"235a0f630ddd0692"}},{"code_sha256_prefix":"e84664190f35139b","entry":"sinkhorn_probs","repo":"pliang279/pid","repo_kind":"official","path":"estimators/ce_alignment_information.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/estimators/ce_alignment_information.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e84664190f35139b"}},{"code_sha256_prefix":"99576a1777b1c03f","entry":"solve_Q","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"99576a1777b1c03f"}},{"code_sha256_prefix":"5a0f6afd9da79b06","entry":"solve_Q_other","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5a0f6afd9da79b06"}},{"code_sha256_prefix":"1eed151775ba41d8","entry":"test","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1eed151775ba41d8"}},{"code_sha256_prefix":"248d63471d1f9b6d","entry":"upper_bound","repo":"pliang279/pid","repo_kind":"official","path":"bounds/bounds.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/bounds/bounds.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"248d63471d1f9b6d"}},{"code_sha256_prefix":"a1dadeb1ac1d82ee","entry":"mlp","repo":"pliang279/pid","repo_kind":"official","path":"estimators/ce_alignment_information.py","file_url":"https://github.com/pliang279/pid/blob/HEAD/estimators/ce_alignment_information.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1dadeb1ac1d82ee"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}