{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sparse-moes-meet-efficient-ensembles","title":"Sparse MoEs meet Efficient Ensembles","arxiv_id":"2110.03360","date":"2021-10-07","proceeding":null,"authors":["James Urquhart Allingham","Florian Wenzel","Zelda E Mariet","Basil Mustafa","Joan Puigcerver","Neil Houlsby","Ghassen Jerfel","Vincent Fortuin","Balaji Lakshminarayanan","Jasper Snoek","Dustin Tran","Carlos Riquelme Ruiz","Rodolphe Jenatton"],"abstract":"Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combination is beneficial. This includes a comprehensive evaluation of sparse MoEs in uncertainty related benchmarks. Then, we present Efficient Ensemble of Experts (E$^3$), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble. Extensive experiments demonstrate the accuracy, log-likelihood, few-shot learning, robustness, and uncertainty improvements of E$^3$ over several challenging vision Transformer-based baselines. E$^3$ not only preserves its efficiency while scaling to models with up to 2.7B parameters, but also provides better predictive performance and uncertainty estimates for larger models.","url_abs":"https://arxiv.org/abs/2110.03360v2","url_pdf":"https://arxiv.org/pdf/2110.03360v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sparse-moes-meet-efficient-ensembles","repo_url":"https://github.com/google-research/vmoe","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"mixture-of-experts","task_name":"Mixture-of-Experts"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2110.03360","atlas_url":"https://app.syntology.ai/?focus=2110.03360","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2110.03360"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/google-research/vmoe","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7f8cef48aff3f297","entry":"get_hyper","repo":"google-research/vmoe","repo_kind":"official","path":"vmoe/configs/eee_paper/eee_s32_last2_ilsvrc2012_ft_cifar100.py","file_url":"https://github.com/google-research/vmoe/blob/HEAD/vmoe/configs/eee_paper/eee_s32_last2_ilsvrc2012_ft_cifar100.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7f8cef48aff3f297"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}