{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/momentumsmoe-integrating-momentum-into-sparse","title":"MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts","arxiv_id":"2410.14574","date":"2024-10-18","proceeding":null,"authors":["Rachel S. Y. Teo","Tan M. Nguyen"],"abstract":"Sparse Mixture of Experts (SMoE) has become the key to unlocking unparalleled scalability in deep learning. SMoE has the potential to exponentially increase parameter count while maintaining the efficiency of the model by only activating a small subset of these parameters for a given sample. However, it has been observed that SMoE suffers from unstable training and has difficulty adapting to new distributions, leading to the model's lack of robustness to data contamination. To overcome these limitations, we first establish a connection between the dynamics of the expert representations in SMoEs and gradient descent on a multi-objective optimization problem. Leveraging our framework, we then integrate momentum into SMoE and propose a new family of SMoEs named MomentumSMoE. We theoretically prove and numerically demonstrate that MomentumSMoE is more stable and robust than SMoE. In particular, we verify the advantages of MomentumSMoE over SMoE on a variety of practical tasks including ImageNet-1K object recognition and WikiText-103 language modeling. We demonstrate the applicability of MomentumSMoE to many types of SMoE models, including those in the Sparse MoE model for vision (V-MoE) and the Generalist Language Model (GLaM). We also show that other advanced momentum-based optimization methods, such as Adam, can be easily incorporated into the MomentumSMoE framework for designing new SMoE models with even better performance, almost negligible additional computation cost, and simple implementations.","url_abs":"https://arxiv.org/abs/2410.14574v1","url_pdf":"https://arxiv.org/pdf/2410.14574v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"momentumsmoe-integrating-momentum-into-sparse","repo_url":"https://github.com/rachtsy/momentumsmoe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"},{"task_slug":"object-recognition","task_name":"Object Recognition"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"moe","method_name":"MoE"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.14574","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.14574"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/facebookresearch/deit","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/google-research/vmoe","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/rachtsy/MomentumSMoE","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/bwconrad/soft-moe","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rachtsy/momentumsmoe","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bafe746746157c90","entry":"cal_mse_loss","repo":"rachtsy/MomentumSMoE","repo_kind":"official","path":"custom_layers_opt.py","file_url":"https://github.com/rachtsy/MomentumSMoE/blob/HEAD/custom_layers_opt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bafe746746157c90"}},{"code_sha256_prefix":"b334bcc524847095","entry":"kl_divergence","repo":"rachtsy/MomentumSMoE","repo_kind":"official","path":"custom_layers_opt.py","file_url":"https://github.com/rachtsy/MomentumSMoE/blob/HEAD/custom_layers_opt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b334bcc524847095"}},{"code_sha256_prefix":"a1a2a52efd68fd82","entry":"pad_sequence_reverse","repo":"rachtsy/MomentumSMoE","repo_kind":"official","path":"finetune_data.py","file_url":"https://github.com/rachtsy/MomentumSMoE/blob/HEAD/finetune_data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1a2a52efd68fd82"}},{"code_sha256_prefix":"65e0b3de6d8d8ac4","entry":"full_eval","repo":"rachtsy/MomentumSMoE","repo_kind":"official","path":"finetune_trainer.py","file_url":"https://github.com/rachtsy/MomentumSMoE/blob/HEAD/finetune_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"65e0b3de6d8d8ac4"}},{"code_sha256_prefix":"d1040ba9a02be2b4","entry":"train_iteration","repo":"rachtsy/MomentumSMoE","repo_kind":"official","path":"finetune_trainer.py","file_url":"https://github.com/rachtsy/MomentumSMoE/blob/HEAD/finetune_trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d1040ba9a02be2b4"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}