Methods › Natural Language Processing › Transformers › Parallel Layers
Parallel Layers
Introduced by Aakanksha Chowdhery et al. in PaLM: Scaling Language Modeling with Pathways
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
• Parallel Layers – We use a “parallel” formulation in each Transformer block (Wang & Komatsuzaki, 2021), rather than the standard “serialized” formulation. Specifically, the standard formulation can be written as:
y = x + MLP(LayerNorm(x + Attention(LayerNorm(x)))
Whereas the parallel formulation can be written as:
y = x + MLP(LayerNorm(x)) + Attention(LayerNorm(x))
The parallel formulation results in roughly 15% faster training speed at large scales, since the MLP and Attention input matrix multiplications can be fused. Ablation experiments showed a small quality degradation at 8B scale but no quality degradation at 62B scale, so we extrapolated that the effect of parallel layers should be quality neutral at the 540B scale.
Papers archive 2025-07-28
3 shown of 3, newest first. Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code; it is per sample and not a correctness claim.
-
Splitformer: An improved early-exit architecture for automatic speech recognition on edge devices 22 Jun 2025 · 1 repository · arXiv:2506.18035
-
Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations 13 Aug 2023 · 0 repositories · arXiv:2308.06679
-
PaLM: Scaling Language Modeling with Pathways 5 Apr 2022 · 7 repositories · arXiv:2204.02311Syntology ran 30 of 37 samples · 7 unverified
Tasks archive 2025-07-28
20 shown of 26 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections