{"url":"/method/parallel-layers","slug":"parallel-layers","name":"Parallel Layers","full_name":"Parallel Layers","full_name_withheld":false,"description_markdown":"• Parallel Layers – We use a “parallel” formulation in each Transformer block (Wang & Komatsuzaki, 2021), rather than the standard “serialized” formulation. Specifically, the standard formulation can be written as:   \r\n    y = x + MLP(LayerNorm(x + Attention(LayerNorm(x)))   \r\n\r\nWhereas the parallel formulation can be written as:   \r\n    y = x + MLP(LayerNorm(x)) + Attention(LayerNorm(x))   \r\n\r\nThe parallel formulation results in roughly 15% faster training speed at large scales, since the MLP and Attention input matrix multiplications can be fused. Ablation experiments showed a small quality degradation at 8B scale but no quality degradation at 62B scale, so we extrapolated that the effect of parallel layers should be quality neutral at the 540B scale.","description_state":"present","introduced_year":null,"introduced_by":{"title":"PaLM: Scaling Language Modeling with Pathways","paper":"/paper/palm-scaling-language-modeling-with-pathways-1","first_author":"Aakanksha Chowdhery","n_authors":67,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/palm-scaling-language-modeling-with-pathways-1"},"source":{"url":"https://arxiv.org/abs/2204.02311v5","title":"PaLM: Scaling Language Modeling with Pathways","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"Natural Language Processing","area_id":"natural-language-processing","collection":"Transformers","url":"/methods/category/transformers","pwc_aliases":[]}],"n_papers_tagged":3,"archive_num_papers":3,"papers_newest_first":[{"paper":"/paper/splitformer-an-improved-early-exit","title":"Splitformer: An improved early-exit architecture for automatic speech recognition on edge devices","date":"2025-06-22","arxiv_id":"2506.18035","n_code_links":1,"syntology":null},{"paper":null,"title":"Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations","date":"2023-08-13","arxiv_id":"2308.06679","n_code_links":0,"syntology":null},{"paper":"/paper/palm-scaling-language-modeling-with-pathways-1","title":"PaLM: Scaling Language Modeling with Pathways","date":"2022-04-05","arxiv_id":"2204.02311","n_code_links":7,"syntology":{"ran":30,"of":37,"unverified":7,"pointer_only":0}}],"papers_shown":3,"tasks":[{"task":"/task/auto-debugging","name":"Auto Debugging","papers":1},{"task":"/task/automatic-speech-recognition-2","name":"Automatic Speech Recognition","papers":1},{"task":"/task/code-generation","name":"Code Generation","papers":1},{"task":"/task/common-sense-reasoning","name":"Common Sense Reasoning","papers":1},{"task":"/task/coreference-resolution","name":"Coreference Resolution","papers":1},{"task":"/task/cross-lingual-question-answering","name":"Cross-Lingual Question Answering","papers":1},{"task":"/task/few-shot-learning","name":"Few-Shot Learning","papers":1},{"task":"/task/hindu-knowledge","name":"Hindu Knowledge","papers":1},{"task":"/task/known-unknowns","name":"Known Unknowns","papers":1},{"task":"/task/language-modeling","name":"Language Modeling","papers":1},{"task":"/task/language-modelling","name":"Language Modelling","papers":1},{"task":"/task/logic-grid-puzzle","name":"Logic Grid Puzzle","papers":1},{"task":"/task/logical-reasoning","name":"Logical Reasoning","papers":1},{"task":"/task/math","name":"Math","papers":1},{"task":"/task/memorization","name":"Memorization","papers":1},{"task":"/task/multi-task-language-understanding","name":"Multi-task Language Understanding","papers":1},{"task":"/task/multiple-choice-qa","name":"Multiple Choice Question Answering (MCQA)","papers":1},{"task":"/task/natural-language-inference","name":"Natural Language Inference","papers":1},{"task":"/task/novel-concepts","name":"Novel Concepts","papers":1},{"task":"/task/question-answering","name":"Question Answering","papers":1}],"tasks_shown":20,"n_tasks":26,"usage_by_year":[{"year":"2022","papers":1},{"year":"2023","papers":1},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/parallel-layers"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}