{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-gating-mechanism-in-sparse-moe","title":"Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gate","arxiv_id":"2505.19525","date":"2025-05-26","proceeding":null,"authors":["Liangwei Nathan Zheng","Wei Emma Zhang","Mingyu Guo","Miao Xu","Olaf Maennel","Weitong Chen"],"abstract":"Effectively managing missing modalities is a fundamental challenge in real-world multimodal learning scenarios, where data incompleteness often results from systematic collection errors or sensor failures. Sparse Mixture-of-Experts (SMoE) architectures have the potential to naturally handle multimodal data, with individual experts specializing in different modalities. However, existing SMoE approach often lacks proper ability to handle missing modality, leading to performance degradation and poor generalization in real-world applications. We propose Conf-SMoE to introduce a two-stage imputation module to handle the missing modality problem for the SMoE architecture and reveal the insight of expert collapse from theoretical analysis with strong empirical evidence. Inspired by our theoretical analysis, Conf-SMoE propose a novel expert gating mechanism by detaching the softmax routing score to task confidence score w.r.t ground truth. This naturally relieves expert collapse without introducing additional load balance loss function. We show that the insights of expert collapse aligns with other gating mechanism such as Gaussian and Laplacian gate. We also evaluate the proposed method on four different real world dataset with three different experiment settings to conduct comprehensive the analysis of Conf-SMoE on modality fusion and resistance to missing modality.","url_abs":"https://arxiv.org/abs/2505.19525v1","url_pdf":"https://arxiv.org/pdf/2505.19525v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-gating-mechanism-in-sparse-moe","repo_url":"https://github.com/icuraslw/official-repository-of-confsmoe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"imputation","task_name":"Imputation"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2505.19525","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.19525"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/icuraslw/official-repository-of-confsmoe","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/zehuiwu/MMML","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/ambisinister/lossfreebalance","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":5},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1},"found_in_text":{"samples":5,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"8cd46b9068cdd1b0","entry":"Attention","repo":"icuraslw/official-repository-of-confsmoe","repo_kind":"official","path":"models.py","file_url":"https://github.com/icuraslw/official-repository-of-confsmoe/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8cd46b9068cdd1b0"}},{"code_sha256_prefix":"4e5ef0a7605c7ce9","entry":"BertCrossattLayer","repo":"zehuiwu/MMML","repo_kind":"found_in_text","path":"utils/cross_attn_encoder.py","file_url":"https://github.com/zehuiwu/MMML/blob/HEAD/utils/cross_attn_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e5ef0a7605c7ce9"}},{"code_sha256_prefix":"10838ca8dd1e6973","entry":"BertIntermediate","repo":"zehuiwu/MMML","repo_kind":"found_in_text","path":"utils/cross_attn_encoder.py","file_url":"https://github.com/zehuiwu/MMML/blob/HEAD/utils/cross_attn_encoder.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"10838ca8dd1e6973"}},{"code_sha256_prefix":"8bd97889b987d97e","entry":"BertAttention","repo":"zehuiwu/MMML","repo_kind":"found_in_text","path":"utils/cross_attn_encoder.py","file_url":"https://github.com/zehuiwu/MMML/blob/HEAD/utils/cross_attn_encoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8bd97889b987d97e"}},{"code_sha256_prefix":"c443f3042c5494ec","entry":"BertSelfattLayer","repo":"zehuiwu/MMML","repo_kind":"found_in_text","path":"utils/cross_attn_encoder.py","file_url":"https://github.com/zehuiwu/MMML/blob/HEAD/utils/cross_attn_encoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c443f3042c5494ec"}},{"code_sha256_prefix":"f860121836e52aad","entry":"CMELayer","repo":"zehuiwu/MMML","repo_kind":"found_in_text","path":"utils/cross_attn_encoder.py","file_url":"https://github.com/zehuiwu/MMML/blob/HEAD/utils/cross_attn_encoder.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f860121836e52aad"}},{"code_sha256_prefix":"f6a076dfda26262a","entry":"ConfSMoE","repo":"icuraslw/official-repository-of-confsmoe","repo_kind":"official","path":"models.py","file_url":"https://github.com/icuraslw/official-repository-of-confsmoe/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6a076dfda26262a"}},{"code_sha256_prefix":"3c8b32bf9f9c0e48","entry":"TransformerEncoderLayer","repo":"icuraslw/official-repository-of-confsmoe","repo_kind":"official","path":"models.py","file_url":"https://github.com/icuraslw/official-repository-of-confsmoe/blob/HEAD/models.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c8b32bf9f9c0e48"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}