{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/unified-normalization-for-accelerating-and","title":"Unified Normalization for Accelerating and Stabilizing Transformers","arxiv_id":"2208.01313","date":"2022-08-02","proceeding":null,"authors":["Qiming Yang","Kai Zhang","Chaoxiang Lan","Zhi Yang","Zheyang Li","Wenming Tan","Jun Xiao","ShiLiang Pu"],"abstract":"Solid results from Transformers have made them prevailing architectures in various natural language and vision tasks. As a default component in Transformers, Layer Normalization (LN) normalizes activations within each token to boost the robustness. However, LN requires on-the-fly statistics calculation in inference as well as division and square root operations, leading to inefficiency on hardware. What is more, replacing LN with other hardware-efficient normalization schemes (e.g., Batch Normalization) results in inferior performance, even collapse in training. We find that this dilemma is caused by abnormal behaviors of activation statistics, including large fluctuations over iterations and extreme outliers across layers. To tackle these issues, we propose Unified Normalization (UN), which can speed up the inference by being fused with other linear operations and achieve comparable performance on par with LN. UN strives to boost performance by calibrating the activation and gradient statistics with a tailored fluctuation smoothing strategy. Meanwhile, an adaptive outlier filtration strategy is applied to avoid collapse in training whose effectiveness is theoretically proved and experimentally verified in this paper. We demonstrate that UN can be an efficient drop-in alternative to LN by conducting extensive experiments on language and vision tasks. Besides, we evaluate the efficiency of our method on GPU. Transformers equipped with UN enjoy about 31% inference speedup and nearly 18% memory reduction. Code will be released at https://github.com/hikvision-research/Unified-Normalization.","url_abs":"https://arxiv.org/abs/2208.01313v1","url_pdf":"https://arxiv.org/pdf/2208.01313v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"unified-normalization-for-accelerating-and","repo_url":"https://github.com/hikvision-research/unified-normalization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2208.01313","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2208.01313"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hikvision-research/unified-normalization","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":6,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3cbdd3b190b16350","entry":"get_mean_and_std","repo":"hikvision-research/unified-normalization","repo_kind":"official","path":"image_classification/utils.py","file_url":"https://github.com/hikvision-research/unified-normalization/blob/HEAD/image_classification/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3cbdd3b190b16350"}},{"code_sha256_prefix":"ced6bc2bfdfe0059","entry":"get_sinusoid_encoding","repo":"hikvision-research/unified-normalization","repo_kind":"official","path":"image_classification/models/transformer_block.py","file_url":"https://github.com/hikvision-research/unified-normalization/blob/HEAD/image_classification/models/transformer_block.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ced6bc2bfdfe0059"}},{"code_sha256_prefix":"5dac11fdc41c896b","entry":"checkpoint_filter_fn","repo":"hikvision-research/unified-normalization","repo_kind":"official","path":"image_classification/models/vit.py","file_url":"https://github.com/hikvision-research/unified-normalization/blob/HEAD/image_classification/models/vit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5dac11fdc41c896b"}},{"code_sha256_prefix":"73f773e3a43253f7","entry":"load_state_dict","repo":"hikvision-research/unified-normalization","repo_kind":"official","path":"image_classification/utils.py","file_url":"https://github.com/hikvision-research/unified-normalization/blob/HEAD/image_classification/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"73f773e3a43253f7"}},{"code_sha256_prefix":"1dbef2d4da4ee10f","entry":"resize_pos_embed","repo":"hikvision-research/unified-normalization","repo_kind":"official","path":"image_classification/models/vit.py","file_url":"https://github.com/hikvision-research/unified-normalization/blob/HEAD/image_classification/models/vit.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1dbef2d4da4ee10f"}},{"code_sha256_prefix":"c08aacf36a89a8f6","entry":"resize_pos_embed","repo":"hikvision-research/unified-normalization","repo_kind":"official","path":"image_classification/utils.py","file_url":"https://github.com/hikvision-research/unified-normalization/blob/HEAD/image_classification/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c08aacf36a89a8f6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}