Methods › General › Intra-Layer Parallel › GShard

GShard

6 papers tagged archive 2025-07-28

Introduced by Dmitry Lepikhin et al. in GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

archive 2025-07-28 Description, source and code snippet are the archive's method entry.

GShard is a intra-layer parallel distributed method. It consists of set of simple APIs for annotations, and a compiler extension in XLA for automatic parallelization.

Paper Source

Papers archive 2025-07-28

6 shown of 6, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
11 Jan 2024 · 2 repositories · arXiv:2401.06066

Syntology ran 6 of 7 samples · 1 unverified
Mixture-of-Experts with Expert Choice Routing
18 Feb 2022 · 0 repositories · arXiv:2202.09368
Scaling End-to-End Models for Large-Scale Multilingual ASR
30 Apr 2021 · 0 repositories · arXiv:2104.14830
Carbon Emissions and Large Neural Network Training
21 Apr 2021 · 0 repositories · arXiv:2104.10350
Compression of Deep Learning Models for Text: A Survey
12 Aug 2020 · 0 repositories · arXiv:2008.05221
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
30 Jun 2020 · 2 repositories · arXiv:2006.16668

Syntology ran 9 of 10 samples · 1 unverified · 1 pointer-only (licence)

Tasks archive 2025-07-28

16 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.

Task	Papers
Mixture-of-Experts	3
Deep Learning	1
Information Retrieval	1
Knowledge Distillation	1
Language Modelling	1
Large Language Model	1
Machine Translation	1
Multi-Task Learning	1
Neural Architecture Search	1
Playing the Game of 2048	1
Quantization	1
Retrieval	1
Scheduling	1
Survey	1
Tensor Decomposition	1
Translation	1

Usage over time archive 2025-07-28

Papers per year the archive tags with this method, by the paper's archive date (6 dated). Bars are counts, not a trend claim.

Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).

Categories archive 2025-07-28

Intra-Layer Parallel Model Parallel Methods Distributed Methods

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections