{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dime-fine-grained-interpretations-of","title":"DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations","arxiv_id":"2203.02013","date":"2022-03-03","proceeding":null,"authors":["Yiwei Lyu","Paul Pu Liang","Zihao Deng","Ruslan Salakhutdinov","Louis-Philippe Morency"],"abstract":"The ability for a human to understand an Artificial Intelligence (AI) model's decision-making process is critical in enabling stakeholders to visualize model behavior, perform model debugging, promote trust in AI models, and assist in collaborative human-AI decision-making. As a result, the research fields of interpretable and explainable AI have gained traction within AI communities as well as interdisciplinary scientists seeking to apply AI in their subject areas. In this paper, we focus on advancing the state-of-the-art in interpreting multimodal models - a class of machine learning methods that tackle core challenges in representing and capturing interactions between heterogeneous data sources such as images, text, audio, and time-series data. Multimodal models have proliferated numerous real-world applications across healthcare, robotics, multimedia, affective computing, and human-computer interaction. By performing model disentanglement into unimodal contributions (UC) and multimodal interactions (MI), our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models while maintaining generality across arbitrary modalities, model architectures, and tasks. Through a comprehensive suite of experiments on both synthetic and real-world multimodal tasks, we show that DIME generates accurate disentangled explanations, helps users of multimodal models gain a deeper understanding of model behavior, and presents a step towards debugging and improving these models for real-world deployment. Code for our experiments can be found at https://github.com/lvyiwei1/DIME.","url_abs":"https://arxiv.org/abs/2203.02013v1","url_pdf":"https://arxiv.org/pdf/2203.02013v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dime-fine-grained-interpretations-of","repo_url":"https://github.com/lvyiwei1/dime","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"disentanglement","task_name":"Disentanglement"},{"task_slug":"time-series-1","task_name":"Time Series"},{"task_slug":"time-series","task_name":"Time Series Analysis"}],"methods":[{"method_slug":"dime","method_name":"DIME"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.02013","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.02013"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lvyiwei1/dime","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"unverified":2},"by_repo_kind":{"official":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e21bb85b8e1b4836","entry":"do_nms","repo":"lvyiwei1/dime","repo_kind":"official","path":"lxmert/modeling_frcnn.py","file_url":"https://github.com/lvyiwei1/dime/blob/HEAD/lxmert/modeling_frcnn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e21bb85b8e1b4836"}},{"code_sha256_prefix":"899d1e2736236351","entry":"is_remote_url","repo":"lvyiwei1/dime","repo_kind":"official","path":"lxmert/utils.py","file_url":"https://github.com/lvyiwei1/dime/blob/HEAD/lxmert/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"899d1e2736236351"}},{"code_sha256_prefix":"8a78a755bda274a5","entry":"load_labels","repo":"lvyiwei1/dime","repo_kind":"official","path":"lxmert/utils.py","file_url":"https://github.com/lvyiwei1/dime/blob/HEAD/lxmert/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8a78a755bda274a5"}},{"code_sha256_prefix":"54a2fff601881b32","entry":"pad_list_tensors","repo":"lvyiwei1/dime","repo_kind":"official","path":"lxmert/modeling_frcnn.py","file_url":"https://github.com/lvyiwei1/dime/blob/HEAD/lxmert/modeling_frcnn.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"54a2fff601881b32"}},{"code_sha256_prefix":"77586e7080d66922","entry":"tryload","repo":"lvyiwei1/dime","repo_kind":"official","path":"lxmert/extracting_data.py","file_url":"https://github.com/lvyiwei1/dime/blob/HEAD/lxmert/extracting_data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"77586e7080d66922"}},{"code_sha256_prefix":"bd3a3f4bd8750da2","entry":"load_checkpoint","repo":"lvyiwei1/dime","repo_kind":"official","path":"lxmert/utils.py","file_url":"https://github.com/lvyiwei1/dime/blob/HEAD/lxmert/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd3a3f4bd8750da2"}},{"code_sha256_prefix":"054bb3d1720e3322","entry":"norm_box","repo":"lvyiwei1/dime","repo_kind":"official","path":"lxmert/modeling_frcnn.py","file_url":"https://github.com/lvyiwei1/dime/blob/HEAD/lxmert/modeling_frcnn.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"054bb3d1720e3322"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}