Mixture-of-Experts

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

No dataset record in the archive lists this task.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 516 papers with code (1,312 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Distilling the Knowledge in a Neural Network
  9 Mar 2015 64 repositories listed Syntology ran 15 of 37 samples · 22 unverified · 10 pointer-only (licence)
  A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.
• Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts
  19 Jul 2018 11 repositories listed
  In this work, we propose a novel multi-task learning approach, Multi-gate Mixture-of-Experts (MMoE), which explicitly learns to model task relationships from data.
• No Language Left Behind: Scaling Human-Centered Machine Translation
  11 Jul 2022 9 repositories listed Syntology ran 0 of 10 samples · 10 unverified
  Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today.
• Gated Multimodal Units for Information Fusion
  7 Feb 2017 9 repositories listed Syntology ran 1 of 1 samples · 0 unverified
  The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different…
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
  11 Jan 2021 8 repositories listed Syntology ran 7 of 13 samples · 6 unverified · 8 pointer-only (licence)
  We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources.
• Qwen2.5 Technical Report
  19 Dec 2024 6 repositories listed Syntology ran 1 of 3 samples · 2 unverified
  In addition, for hosted solutions, the proprietary models currently include two mixture-of-experts (MoE) variants: Qwen2.
• Qwen2 Technical Report
  15 Jul 2024 6 repositories listed
  This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models.
• Mixtral of Experts
  8 Jan 2024 6 repositories listed Syntology ran 5 of 5 samples · 0 unverified
  In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks.
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
  7 May 2024 5 repositories listed Syntology ran 2 of 3 samples · 1 unverified
  MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation.
• JetMoE: Reaching Llama2 Performance with 0.1M Dollars
  11 Apr 2024 5 repositories listed Syntology ran 2 of 3 samples · 1 unverified · 1 pointer-only (licence)
  Large Language Models (LLMs) have achieved remarkable results, but their increasing resource demand has become a major obstacle to the development of powerful and accessible super-human intelligence.
• Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts
  27 Feb 2025 4 repositories listed Syntology ran 1 of 20 samples · 19 unverified
  The inter-device communication of a MoE layer can occupy 47% time of the entire model execution with popular models and frameworks.
• DeepSeek-V3 Technical Report
  27 Dec 2024 4 repositories listed Syntology ran 0 of 7 samples · 7 unverified
  We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token.
• Unity by Diversity: Improved Representation Learning in Multimodal VAEs
  8 Mar 2024 4 repositories listed Syntology ran 8 of 9 samples · 1 unverified · 4 pointer-only (licence)
  Such architectures impose hard constraints on the model.
• Fast Feedforward Networks
  28 Aug 2023 4 repositories listed
  We break the linear link between the layer size and its inference cost by introducing the fast feedforward (FFF) architecture, a log-time alternative to feedforward networks.
• Long-Tailed Visual Recognition via Self-Heterogeneous Integration with Knowledge Excavation
  3 Apr 2023 4 repositories listed Syntology ran 5 of 7 samples · 2 unverified · 7 pointer-only (licence)
  To address this problem, state-of-the-art methods usually adopt a mixture of experts (MoE) to focus on different parts of the long-tailed distribution.
• Robust Federated Learning by Mixture of Experts
  23 Apr 2021 4 repositories listed
  We present a novel weighted average model based on the mixture of experts (MoE) concept to provide robustness in Federated learning (FL) against the poisoned/corrupted/outdated local models.
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
  23 Jan 2017 4 repositories listed Syntology ran 4 of 6 samples · 2 unverified · 6 pointer-only (licence)
  In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on…
• Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent
  4 Nov 2024 3 repositories listed
  In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of…
• MoH: Multi-Head Attention as Mixture-of-Head Attention
  15 Oct 2024 3 repositories listed Syntology ran 14 of 18 samples · 4 unverified · 3 pointer-only (licence)
  We show that multi-head attention can be expressed in the summation form.
• MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
  9 Oct 2024 3 repositories listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  In this work, we aim to simultaneously enhance the effectiveness and efficiency of Mixture-of-Experts (MoE) methods.
• Jamba: A Hybrid Transformer-Mamba Language Model
  28 Mar 2024 3 repositories listed
  We present Jamba, a new base large language model based on a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture.
• Scattered Mixture-of-Experts Implementation
  13 Mar 2024 3 repositories listed Syntology ran 1 of 3 samples · 2 unverified
  We present ScatterMoE, an implementation of Sparse Mixture-of-Experts (SMoE) on GPUs.
• Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral
  4 Mar 2024 3 repositories listed
  Mixtral, a representative sparse mixture of experts (SMoE) language model, has received significant attention due to its unique model design and superior performance.
• MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
  29 Jan 2024 3 repositories listed Syntology ran 0 of 2 samples · 2 unverified · 2 pointer-only (licence)
  In this work, we propose a simple yet effective training strategy MoE-Tuning for LVLMs.
• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
  29 Nov 2022 3 repositories listed Syntology ran 1 of 6 samples · 5 unverified
  We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs.
• ST-MoE: Designing Stable and Transferable Sparse Expert Models
  17 Feb 2022 3 repositories listed Syntology ran 5 of 5 samples · 0 unverified · 5 pointer-only (licence)
  But advancing the state-of-the-art across a broad set of natural language tasks has been hindered by training instabilities and uncertain quality during fine-tuning.
• DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
  14 Jan 2022 3 repositories listed
  As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most promising model architectures due to…
• DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning
  7 Jun 2021 3 repositories listed Syntology ran 1 of 9 samples · 8 unverified
  State-of-the-art MoE models use a trainable sparse gate to select a subset of the experts for each input example.
• Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models
  8 Nov 2019 3 repositories listed Syntology ran 1 of 8 samples · 7 unverified · 1 pointer-only (licence)
  In this work, we characterise successful learning of such models as the fulfillment of four criteria: i) implicit latent decomposition into shared and private subspaces, ii) coherent joint generation over all…
• Non-Normal Mixtures of Experts
  22 Jun 2015 3 repositories listed
  Mixture of Experts (MoE) is a popular framework for modeling heterogeneity in data for regression, classification and clustering.

Syntology lines on 21 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.