{"url":"/method/layerscale","slug":"layerscale","name":"LayerScale","full_name":"LayerScale","full_name_withheld":false,"description_markdown":"**LayerScale** is a method used for [vision transformer](https://paperswithcode.com/methods/category/vision-transformer) architectures to help improve training dynamics. It adds a learnable diagonal matrix on output of each residual block, initialized close to (but not at) 0. Adding this simple layer after each residual block improves the training dynamic, allowing for the training of deeper high-capacity image transformers that benefit from depth.\r\n\r\nSpecifically, LayerScale is a per-channel multiplication of the vector produced by each residual block, as opposed to a single scalar, see Figure (d). The objective is to group the updates of the weights associated with the same output channel. Formally, LayerScale is a multiplication by a diagonal matrix on output of each residual block. In other words:\r\n\r\n$$\r\nx\\_{l}^{\\prime} =x\\_{l}+\\operatorname{diag}\\left(\\lambda\\_{l, 1}, \\ldots, \\lambda\\_{l, d}\\right) \\times \\operatorname{SA}\\left(\\eta\\left(x\\_{l}\\right)\\right) \r\n$$\r\n\r\n$$\r\nx\\_{l+1} =x\\_{l}^{\\prime}+\\operatorname{diag}\\left(\\lambda\\_{l, 1}^{\\prime}, \\ldots, \\lambda\\_{l, d}^{\\prime}\\right) \\times \\operatorname{FFN}\\left(\\eta\\left(x\\_{l}^{\\prime}\\right)\\right)\r\n$$\r\n\r\nwhere the parameters $\\lambda\\_{l, i}$ and $\\lambda\\_{l, i}^{\\prime}$ are learnable weights. The diagonal values are all initialized to a fixed small value $\\varepsilon:$ we set it to $\\varepsilon=0.1$ until depth 18 , $\\varepsilon=10^{-5}$ for depth 24 and $\\varepsilon=10^{-6}$ for deeper networks. \r\n\r\nThis formula is akin to other [normalization](https://paperswithcode.com/methods/category/normalization) strategies [ActNorm](https://paperswithcode.com/method/activation-normalization) or [LayerNorm](https://paperswithcode.com/method/layer-normalization) but executed on output of the residual block. Yet LayerScale seeks a different effect: [ActNorm](https://paperswithcode.com/method/activation-normalization) is a data-dependent initialization that calibrates activations so that they have zero-mean and unit variance, like [BatchNorm](https://paperswithcode.com/method/batch-normalization). In contrast, in LayerScale, we initialize the diagonal with small values so that the initial contribution of the residual branches to the function implemented by the transformer is small. In that respect the motivation is therefore closer to that of [ReZero](https://paperswithcode.com/method/rezero), [SkipInit](https://paperswithcode.com/method/skipinit), [Fixup](https://paperswithcode.com/method/fixup-initialization) and [T-Fixup](https://paperswithcode.com/method/t-fixup): to train closer to the identity function and let the network integrate the additional parameters progressively during the training. LayerScale offers more diversity in the optimization than just adjusting the whole layer by a single learnable scalar as in [ReZero](https://paperswithcode.com/method/rezero)/[SkipInit](https://paperswithcode.com/method/skipinit), [Fixup](https://paperswithcode.com/method/fixup-initialization) and [T-Fixup](https://paperswithcode.com/method/t-fixup).","description_state":"present","introduced_year":null,"introduced_by":{"title":"Going deeper with Image Transformers","paper":"/paper/going-deeper-with-image-transformers","first_author":"Hugo Touvron","n_authors":5,"url_abs":null,"archive_paper_url":"https://paperswithcode.com/paper/going-deeper-with-image-transformers"},"source":{"url":"https://arxiv.org/abs/2103.17239v2","title":"Going deeper with Image Transformers","url_on_a_paper_host":true},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Normalization","url":"/methods/category/normalization","pwc_aliases":[]},{"area":"General","area_id":"general","collection":"Regularization","url":"/methods/category/regularization","pwc_aliases":[]}],"n_papers_tagged":12,"archive_num_papers":12,"papers_newest_first":[{"paper":"/paper/guiding-the-experts-semantic-priors-for","title":"Guiding the Experts: Semantic Priors for Efficient and Focused MoE Routing","date":"2025-05-24","arxiv_id":"2505.18586","n_code_links":1,"syntology":null},{"paper":"/paper/spectralkd-understanding-and-optimizing","title":"SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis","date":"2024-12-26","arxiv_id":"2412.19055","n_code_links":1,"syntology":null},{"paper":null,"title":"Detecting Severity of Diabetic Retinopathy from Fundus Images: A Transformer Network-based Review","date":"2023-01-03","arxiv_id":"2301.00973","n_code_links":0,"syntology":null},{"paper":null,"title":"MaiT: Leverage Attention Masks for More Efficient Image Transformers","date":"2022-07-06","arxiv_id":"2207.03006","n_code_links":0,"syntology":null},{"paper":"/paper/deit-iii-revenge-of-the-vit","title":"DeiT III: Revenge of the ViT","date":"2022-04-14","arxiv_id":"2204.07118","n_code_links":12,"syntology":null},{"paper":"/paper/three-things-everyone-should-know-about","title":"Three things everyone should know about Vision Transformers","date":"2022-03-18","arxiv_id":"2203.09795","n_code_links":8,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}},{"paper":"/paper/a-convnet-for-the-2020s","title":"A ConvNet for the 2020s","date":"2022-01-10","arxiv_id":"2201.03545","n_code_links":54,"syntology":{"ran":54,"of":80,"unverified":26,"pointer_only":11}},{"paper":"/paper/augmenting-convolutional-networks-with","title":"Augmenting Convolutional networks with attention-based aggregation","date":"2021-12-27","arxiv_id":"2112.13692","n_code_links":5,"syntology":{"ran":1,"of":2,"unverified":1,"pointer_only":0}},{"paper":null,"title":"Poformer: A simple pooling transformer for speaker verification","date":"2021-10-10","arxiv_id":"2110.04692","n_code_links":0,"syntology":null},{"paper":"/paper/mait-integrating-spatial-locality-into-image","title":"MaiT: integrating spatial locality into image transformers with attention masks","date":"2021-09-29","arxiv_id":null,"n_code_links":1,"syntology":null},{"paper":"/paper/resmlp-feedforward-networks-for-image","title":"ResMLP: Feedforward networks for image classification with data-efficient training","date":"2021-05-07","arxiv_id":"2105.03404","n_code_links":19,"syntology":{"ran":2,"of":7,"unverified":5,"pointer_only":0}},{"paper":"/paper/going-deeper-with-image-transformers","title":"Going deeper with Image Transformers","date":"2021-03-31","arxiv_id":"2103.17239","n_code_links":21,"syntology":{"ran":5,"of":11,"unverified":6,"pointer_only":2}}],"papers_shown":12,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":6},{"task":"/task/image-classification","name":"image-classification","papers":5},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":3},{"task":"/task/transfer-learning","name":"Transfer Learning","papers":3},{"task":"/task/classification-1","name":"Classification","papers":2},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":2},{"task":"/task/fine-grained-image-classification","name":"Fine-Grained Image Classification","papers":2},{"task":"/task/object-detection","name":"Object Detection","papers":2},{"task":"/task/domain-generalization","name":"Domain Generalization","papers":1},{"task":"/task/classification","name":"General Classification","papers":1},{"task":"/task/image-segmentation","name":"Image Segmentation","papers":1},{"task":"/task/knowledge-distillation","name":"Knowledge Distillation","papers":1},{"task":"/task/machine-translation","name":"Machine Translation","papers":1},{"task":"/task/mixture-of-experts","name":"Mixture-of-Experts","papers":1},{"task":"/task/real-time-object-detection","name":"Real-Time Object Detection","papers":1},{"task":"/task/self-supervised-image-classification","name":"Self-Supervised Image Classification","papers":1},{"task":"/task/self-supervised-learning","name":"Self-Supervised Learning","papers":1},{"task":"/task/speaker-verification","name":"Speaker Verification","papers":1},{"task":"/task/translation","name":"Translation","papers":1}],"tasks_shown":19,"n_tasks":19,"usage_by_year":[{"year":"2021","papers":5},{"year":"2022","papers":4},{"year":"2023","papers":1},{"year":"2024","papers":1},{"year":"2025","papers":1}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/layerscale"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}