{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-14521","title":"Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm","arxiv_id":"2605.14521","date":"2026-05-14","proceeding":null,"authors":["Yuxin Guo","Yihao Yue","Yunhao Ni","Yizhou Ruan","Jie Luo","Wenjun Wu","Lei Huang"],"abstract":"Layer normalization (LN) is a fundamental component in modern deep learning, but its per-sample centering and scaling introduce non-negligible inference overhead. RMSNorm improves efficiency by removing the centering operation, yet this may discard benefits associated with centering. This paper propose a framework to determine whether an LN in an arbitrary DNN can be replaced by RMSNorm without changing the model function. The key idea is to fold LN's centering operation into upstream general linear layers by enforcing zero-mean outputs through the column-centered constraint (CCC) and column-based weight centering (CBWC). We extend the analysis to arbitrary DNNs, define such LNs as foldable LNs, and develop a graph-based detection algorithm. Our analysis shows that many LNs in widely used architectures are foldable, enabling exact inference-time conversion and end-to-end acceleration of 2% to 12% without changing model predictions. Experiments across multiple task families further show that, when exact equivalence is partially broken in practical training settings, our method remains competitive with vanilla LN while improving efficiency.","url_abs":"https://arxiv.org/abs/2605.14521","url_pdf":"https://arxiv.org/pdf/2605.14521","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.14521","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.14521"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/BobYue-01/Enjoy-LN","reach":null}],"summary":{"ran":3,"unverified":6},"by_repo_kind":{"found_in_text":{"samples":9,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"971ebfd67f2e9e7c","entry":"create_analyse_hook_fns","repo":"BobYue-01/Enjoy-LN","repo_kind":"found_in_text","path":"src/enjoyln/utils/hook.py","file_url":"https://github.com/BobYue-01/Enjoy-LN/blob/HEAD/src/enjoyln/utils/hook.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"971ebfd67f2e9e7c"}},{"code_sha256_prefix":"a88d0e250fdc14c4","entry":"fail_str","repo":"BobYue-01/Enjoy-LN","repo_kind":"found_in_text","path":"exps/utils/checker.py","file_url":"https://github.com/BobYue-01/Enjoy-LN/blob/HEAD/exps/utils/checker.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a88d0e250fdc14c4"}},{"code_sha256_prefix":"4c7d8d6ae4adced3","entry":"native_rms_norm_forward","repo":"BobYue-01/Enjoy-LN","repo_kind":"found_in_text","path":"src/enjoyln/modules/rms_norm.py","file_url":"https://github.com/BobYue-01/Enjoy-LN/blob/HEAD/src/enjoyln/modules/rms_norm.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c7d8d6ae4adced3"}},{"code_sha256_prefix":"3ee7b2b1ca74bb3b","entry":"center_conv1d","repo":"BobYue-01/Enjoy-LN","repo_kind":"found_in_text","path":"src/enjoyln/modules/cc_conv1d.py","file_url":"https://github.com/BobYue-01/Enjoy-LN/blob/HEAD/src/enjoyln/modules/cc_conv1d.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3ee7b2b1ca74bb3b"}},{"code_sha256_prefix":"ee23c6ad5582d62d","entry":"center_embedding","repo":"BobYue-01/Enjoy-LN","repo_kind":"found_in_text","path":"src/enjoyln/modules/cc_embedding.py","file_url":"https://github.com/BobYue-01/Enjoy-LN/blob/HEAD/src/enjoyln/modules/cc_embedding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ee23c6ad5582d62d"}},{"code_sha256_prefix":"63c553676a72d007","entry":"center_linear","repo":"BobYue-01/Enjoy-LN","repo_kind":"found_in_text","path":"src/enjoyln/modules/cc_linear.py","file_url":"https://github.com/BobYue-01/Enjoy-LN/blob/HEAD/src/enjoyln/modules/cc_linear.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"63c553676a72d007"}},{"code_sha256_prefix":"599781c677c9a586","entry":"center_modules","repo":"BobYue-01/Enjoy-LN","repo_kind":"found_in_text","path":"src/enjoyln/modules/cc_modules.py","file_url":"https://github.com/BobYue-01/Enjoy-LN/blob/HEAD/src/enjoyln/modules/cc_modules.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"599781c677c9a586"}},{"code_sha256_prefix":"33a0026a8d2f07f0","entry":"fmt_str","repo":"BobYue-01/Enjoy-LN","repo_kind":"found_in_text","path":"exps/utils/checker.py","file_url":"https://github.com/BobYue-01/Enjoy-LN/blob/HEAD/exps/utils/checker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33a0026a8d2f07f0"}},{"code_sha256_prefix":"bd319fa455da4d21","entry":"pass_str","repo":"BobYue-01/Enjoy-LN","repo_kind":"found_in_text","path":"exps/utils/checker.py","file_url":"https://github.com/BobYue-01/Enjoy-LN/blob/HEAD/exps/utils/checker.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd319fa455da4d21"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}