{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/competesmoe-statistically-guaranteed-mixture","title":"CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition","arxiv_id":"2505.13380","date":"2025-05-19","proceeding":null,"authors":["Nam V. Nguyen","Huy Nguyen","Quang Pham","Van Nguyen","Savitha Ramasamy","Nhat Ho"],"abstract":"Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, we argue that effective SMoE training remains challenging because of the suboptimal routing process where experts that perform computation do not directly contribute to the routing process. In this work, we propose competition, a novel mechanism to route tokens to experts with the highest neural response. Theoretically, we show that the competition mechanism enjoys a better sample efficiency than the traditional softmax routing. Furthermore, we develop CompeteSMoE, a simple yet effective algorithm to train large language models by deploying a router to learn the competition policy, thus enjoying strong performances at a low training overhead. Our extensive empirical evaluations on both the visual instruction tuning and language pre-training tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE compared to state-of-the-art SMoE strategies. We have made the implementation available at: https://github.com/Fsoft-AIC/CompeteSMoE. This work is an improved version of the previous study at arXiv:2402.02526","url_abs":"https://arxiv.org/abs/2505.13380v1","url_pdf":"https://arxiv.org/pdf/2505.13380v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"competesmoe-statistically-guaranteed-mixture","repo_url":"https://github.com/fsoft-aic/competesmoe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"competesmoe-statistically-guaranteed-mixture","repo_url":"https://github.com/giangdip2410/competesmoe","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2505.13380","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.13380"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/fsoft-aic/competesmoe","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/giangdip2410/competesmoe","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":3,"ran_draft_wrong":3,"unverified":2},"by_repo_kind":{"official":{"samples":2,"ran":2,"repositories":1},"listed":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"bafe746746157c90","entry":"cal_mse_loss","repo":"giangdip2410/competesmoe","repo_kind":"listed","path":"custom_layers_opt.py","file_url":"https://github.com/giangdip2410/competesmoe/blob/HEAD/custom_layers_opt.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bafe746746157c90"}},{"code_sha256_prefix":"b05e3f5d170026a2","entry":"get_cache_dir","repo":"fsoft-aic/competesmoe","repo_kind":"official","path":"evaluate/lmms_eval/tasks/_task_utils/video_loader.py","file_url":"https://github.com/fsoft-aic/competesmoe/blob/HEAD/evaluate/lmms_eval/tasks/_task_utils/video_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b05e3f5d170026a2"}},{"code_sha256_prefix":"7ac7deae0a636b3f","entry":"get_video","repo":"fsoft-aic/competesmoe","repo_kind":"official","path":"evaluate/lmms_eval/tasks/_task_utils/video_loader.py","file_url":"https://github.com/fsoft-aic/competesmoe/blob/HEAD/evaluate/lmms_eval/tasks/_task_utils/video_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7ac7deae0a636b3f"}},{"code_sha256_prefix":"ea06eaae4fc1eaf0","entry":"hash_args","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"ea06eaae4fc1eaf0"}},{"code_sha256_prefix":"b334bcc524847095","entry":"kl_divergence","repo":"giangdip2410/competesmoe","repo_kind":"listed","path":"custom_layers_opt.py","file_url":"https://github.com/giangdip2410/competesmoe/blob/HEAD/custom_layers_opt.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b334bcc524847095"}},{"code_sha256_prefix":"a1a2a52efd68fd82","entry":"pad_sequence_reverse","repo":"giangdip2410/competesmoe","repo_kind":"listed","path":"finetune_data.py","file_url":"https://github.com/giangdip2410/competesmoe/blob/HEAD/finetune_data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a1a2a52efd68fd82"}},{"code_sha256_prefix":"65e0b3de6d8d8ac4","entry":"full_eval","repo":"giangdip2410/competesmoe","repo_kind":"listed","path":"finetune_trainer.py","file_url":"https://github.com/giangdip2410/competesmoe/blob/HEAD/finetune_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65e0b3de6d8d8ac4"}},{"code_sha256_prefix":"ff471c763eaed63c","entry":"train_iteration","repo":"giangdip2410/competesmoe","repo_kind":"listed","path":"finetune_trainer.py","file_url":"https://github.com/giangdip2410/competesmoe/blob/HEAD/finetune_trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ff471c763eaed63c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}