{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/robust-mixture-of-expert-training-for","title":"Robust Mixture-of-Expert Training for Convolutional Neural Networks","arxiv_id":"2308.10110","date":"2023-08-19","proceeding":"ICCV 2023 1","authors":["Yihua Zhang","Ruisi Cai","Tianlong Chen","Guanhua Zhang","huan zhang","Pin-Yu Chen","Shiyu Chang","Zhangyang Wang","Sijia Liu"],"abstract":"Sparsely-gated Mixture of Expert (MoE), an emerging deep model architecture, has demonstrated a great promise to enable high-accuracy and ultra-efficient model inference. Despite the growing popularity of MoE, little work investigated its potential to advance convolutional neural networks (CNNs), especially in the plane of adversarial robustness. Since the lack of robustness has become one of the main hurdles for CNNs, in this paper we ask: How to adversarially robustify a CNN-based MoE model? Can we robustly train it like an ordinary CNN model? Our pilot study shows that the conventional adversarial training (AT) mechanism (developed for vanilla CNNs) no longer remains effective to robustify an MoE-CNN. To better understand this phenomenon, we dissect the robustness of an MoE-CNN into two dimensions: Robustness of routers (i.e., gating functions to select data-specific experts) and robustness of experts (i.e., the router-guided pathways defined by the subnetworks of the backbone CNN). Our analyses show that routers and experts are hard to adapt to each other in the vanilla AT. Thus, we propose a new router-expert alternating Adversarial training framework for MoE, termed AdvMoE. The effectiveness of our proposal is justified across 4 commonly-used CNN model architectures over 4 benchmark datasets. We find that AdvMoE achieves 1% ~ 4% adversarial robustness improvement over the original dense CNN, and enjoys the efficiency merit of sparsity-gated MoE, leading to more than 50% inference cost reduction. Codes are available at https://github.com/OPTML-Group/Robust-MoE-CNN.","url_abs":"https://arxiv.org/abs/2308.10110v1","url_pdf":"https://arxiv.org/pdf/2308.10110v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"robust-mixture-of-expert-training-for","repo_url":"https://github.com/optml-group/robust-moe-cnn","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"adversarial-robustness","task_name":"Adversarial Robustness"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2308.10110","atlas_url":"https://app.syntology.ai/?focus=2308.10110","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2308.10110"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/optml-group/robust-moe-cnn","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/OPTML-Group/Robust-MoE-CNN","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":1},"by_repo_kind":{"official":{"samples":7,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"0caa8e17b9ed30fc","entry":"GetMask","repo":"OPTML-Group/Robust-MoE-CNN","repo_kind":"official","path":"models/layers/moe_layer.py","file_url":"https://github.com/OPTML-Group/Robust-MoE-CNN/blob/HEAD/models/layers/moe_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0caa8e17b9ed30fc"}},{"code_sha256_prefix":"a9a9b6c8b99e6502","entry":"MoEBase","repo":"OPTML-Group/Robust-MoE-CNN","repo_kind":"official","path":"models/layers/moe_layer.py","file_url":"https://github.com/OPTML-Group/Robust-MoE-CNN/blob/HEAD/models/layers/moe_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a9a9b6c8b99e6502"}},{"code_sha256_prefix":"8ed16518c0657888","entry":"MoEConv","repo":"OPTML-Group/Robust-MoE-CNN","repo_kind":"official","path":"models/layers/moe_layer.py","file_url":"https://github.com/OPTML-Group/Robust-MoE-CNN/blob/HEAD/models/layers/moe_layer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8ed16518c0657888"}},{"code_sha256_prefix":"c79cb52097b0f34e","entry":"conv1x1","repo":"optml-group/robust-moe-cnn","repo_kind":"official","path":"models/resnet_imagenet_moe.py","file_url":"https://github.com/optml-group/robust-moe-cnn/blob/HEAD/models/resnet_imagenet_moe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c79cb52097b0f34e"}},{"code_sha256_prefix":"eaf219fab088a30d","entry":"conv3x3","repo":"optml-group/robust-moe-cnn","repo_kind":"official","path":"models/resnet_imagenet_moe.py","file_url":"https://github.com/optml-group/robust-moe-cnn/blob/HEAD/models/resnet_imagenet_moe.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"eaf219fab088a30d"}},{"code_sha256_prefix":"04c97ef33fd4a1b7","entry":"get_device","repo":"optml-group/robust-moe-cnn","repo_kind":"official","path":"models/layers/moe_layer.py","file_url":"https://github.com/optml-group/robust-moe-cnn/blob/HEAD/models/layers/moe_layer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"04c97ef33fd4a1b7"}},{"code_sha256_prefix":"3b99baf0c1a4b7f2","entry":"percentile","repo":"optml-group/robust-moe-cnn","repo_kind":"official","path":"models/resnet_cifar_moe.py","file_url":"https://github.com/optml-group/robust-moe-cnn/blob/HEAD/models/resnet_cifar_moe.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b99baf0c1a4b7f2"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}