{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/flex-moe-modeling-arbitrary-modality","title":"Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts","arxiv_id":"2410.08245","date":"2024-10-10","proceeding":null,"authors":["Sukwon Yun","Inyoung Choi","Jie Peng","Yangfan Wu","Jingxuan Bao","Qiyiwen Zhang","Jiayi Xin","Qi Long","Tianlong Chen"],"abstract":"Multimodal learning has gained increasing importance across various fields, offering the ability to integrate data from diverse sources such as images, text, and personalized records, which are frequently observed in medical domains. However, in scenarios where some modalities are missing, many existing frameworks struggle to accommodate arbitrary modality combinations, often relying heavily on a single modality or complete data. This oversight of potential modality combinations limits their applicability in real-world situations. To address this challenge, we propose Flex-MoE (Flexible Mixture-of-Experts), a new framework designed to flexibly incorporate arbitrary modality combinations while maintaining robustness to missing data. The core idea of Flex-MoE is to first address missing modalities using a new missing modality bank that integrates observed modality combinations with the corresponding missing ones. This is followed by a uniquely designed Sparse MoE framework. Specifically, Flex-MoE first trains experts using samples with all modalities to inject generalized knowledge through the generalized router ($\\mathcal{G}$-Router). The $\\mathcal{S}$-Router then specializes in handling fewer modality combinations by assigning the top-1 gate to the expert corresponding to the observed modality combination. We evaluate Flex-MoE on the ADNI dataset, which encompasses four modalities in the Alzheimer's Disease domain, as well as on the MIMIC-IV dataset. The results demonstrate the effectiveness of Flex-MoE highlighting its ability to model arbitrary modality combinations in diverse missing modality scenarios. Code is available at https://github.com/UNITES-Lab/flex-moe.","url_abs":"https://arxiv.org/abs/2410.08245v2","url_pdf":"https://arxiv.org/pdf/2410.08245v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"flex-moe-modeling-arbitrary-modality","repo_url":"https://github.com/unites-lab/flex-moe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"moe","method_name":"MoE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.08245","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.08245"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/UNITES-Lab/flex-moe","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/unites-lab/flex-moe","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_violates":1,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"40e694fe9ff5b31a","entry":"TransformerEncoderLayer","repo":"unites-lab/flex-moe","repo_kind":"official","path":"models.py","file_url":"https://github.com/unites-lab/flex-moe/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"40e694fe9ff5b31a"}},{"code_sha256_prefix":"9cc84aa4e66897c2","entry":"convert_ids_to_index","repo":"UNITES-Lab/Flex-MoE","repo_kind":"official","path":"data.py","file_url":"https://github.com/UNITES-Lab/Flex-MoE/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9cc84aa4e66897c2"}},{"code_sha256_prefix":"6b86acff8a357c55","entry":"setup_logger","repo":"UNITES-Lab/flex-moe","repo_kind":"official","path":"utils.py","file_url":"https://github.com/UNITES-Lab/flex-moe/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6b86acff8a357c55"}},{"code_sha256_prefix":"fadc889a6ef30337","entry":"FlexMoE","repo":"unites-lab/flex-moe","repo_kind":"official","path":"models.py","file_url":"https://github.com/unites-lab/flex-moe/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fadc889a6ef30337"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}