{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/root-mean-square-layer-normalization","title":"Root Mean Square Layer Normalization","arxiv_id":"1910.07467","date":"2019-10-16","proceeding":"NeurIPS 2019 12","authors":["Biao Zhang","Rico Sennrich"],"abstract":"Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and weight matrix. However, the computational overhead introduced by LayerNorm makes these improvements expensive and significantly slows the underlying network, e.g. RNN in particular. In this paper, we hypothesize that re-centering invariance in LayerNorm is dispensable and propose root mean square layer normalization, or RMSNorm. RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm. We also present partial RMSNorm, or pRMSNorm where the RMS is estimated from p% of the summed inputs without breaking the above properties. Extensive experiments on several tasks using diverse network architectures show that RMSNorm achieves comparable performance against LayerNorm but reduces the running time by 7%~64% on different models. Source code is available at https://github.com/bzhangGo/rmsnorm.","url_abs":"https://arxiv.org/abs/1910.07467v1","url_pdf":"https://arxiv.org/pdf/1910.07467v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"root-mean-square-layer-normalization","repo_url":"https://github.com/bzhangGo/rmsnorm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"root-mean-square-layer-normalization","repo_url":"https://github.com/hazdzz/RMSNorm","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"root-mean-square-layer-normalization","repo_url":"https://github.com/mirage-project/mirage","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"root-mean-square-layer-normalization","repo_url":"https://github.com/MindCode-4/code-13/tree/main/root-mean-square-layer-normalization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"root-mean-square-layer-normalization","repo_url":"https://github.com/MindCode-4/code-8/tree/main/root-mean-square-layer-normalization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"root-mean-square-layer-normalization","repo_url":"https://github.com/MindCode-4/code-9/tree/main/root-mean-square-layer-normalization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"root-mean-square-layer-normalization","repo_url":"https://github.com/MindSpore-scientific/code-14/tree/main/root-mean-square-layer-normalization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"root-mean-square-layer-normalization","repo_url":"https://github.com/MindSpore-scientific/code-5/tree/main/root-mean-square-layer-normalization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null}],"tasks":[],"methods":[{"method_slug":"rmsnorm","method_name":"RMSNorm"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1910.07467","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1910.07467"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-9/tree/main/root-mean-square-layer-normalization","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-14/tree/main/root-mean-square-layer-normalization","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-8/tree/main/root-mean-square-layer-normalization","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific/code-5/tree/main/root-mean-square-layer-normalization","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mirage-project/mirage","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazdzz/RMSNorm","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-13/tree/main/root-mean-square-layer-normalization","reach":null},{"provenance":"deterministic:regex_extraction","url":"https://github.com/bzhangGo/rmsnorm","reach":null}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9c235f0c904962f7","entry":"RMSNorm","repo":"hazdzz/RMSNorm","repo_kind":"listed","path":"norm.py","file_url":"https://github.com/hazdzz/RMSNorm/blob/HEAD/norm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c235f0c904962f7"}},{"code_sha256_prefix":"9141d54932b3c458","entry":"rms_norm","repo":"bzhangGo/rmsnorm","repo_kind":"official","path":"rmsnorm_tensorflow.py","file_url":"https://github.com/bzhangGo/rmsnorm/blob/HEAD/rmsnorm_tensorflow.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9141d54932b3c458"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}