Methods › General › Mixture-of-Experts › DSelect-k
DSelect-k
Introduced by Hussein Hazimeh et al. in DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
DSelect-k is a continuously differentiable and sparse gate for Mixture-of-experts (MoE), based on a novel binary encoding formulation. Given a user-specified parameter k, the gate selects at most k out of the n experts. The gate can be trained using first-order methods, such as stochastic gradient descent, and offers explicit control over the number of experts to select. This explicit control over sparsity leads to a cardinality-constrained optimization problem, which is computationally challenging. To circumvent this challenge, the authors use a unconstrained reformulation that is equivalent to the original problem. The reformulated problem uses a binary encoding scheme to implicitly enforce the cardinality constraint. By carefully smoothing the binary encoding variables, the reformulated problem can be effectively optimized using first-order methods such as SGD.
The motivation for this method is that existing sparse gates, such as Top-k, are not smooth. The lack of smoothness can lead to convergence and statistical performance issues when training with gradient-based methods.
Papers archive 2025-07-28
2 shown of 2, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Search 5 Jun 2023 · 2 repositories · arXiv:2306.02824
-
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning 7 Jun 2021 · 3 repositories · arXiv:2106.03760Syntology ran 1 of 9 samples · 8 unverified
Tasks archive 2025-07-28
5 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
| Task | Papers |
|---|---|
| Mixture-of-Experts | 2 |
| Recommendation Systems | 2 |
| Language Modeling | 1 |
| Language Modelling | 1 |
| Multi-Task Learning | 1 |
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections