{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fixup-initialization-residual-learning","title":"Fixup Initialization: Residual Learning Without Normalization","arxiv_id":"1901.09321","date":"2019-01-27","proceeding":"ICLR 2019 5","authors":["Hongyi Zhang","Yann N. Dauphin","Tengyu Ma"],"abstract":"Normalization layers are a staple in state-of-the-art deep neural network\narchitectures. They are widely believed to stabilize training, enable higher\nlearning rate, accelerate convergence and improve generalization, though the\nreason for their effectiveness is still an active research topic. In this work,\nwe challenge the commonly-held beliefs by showing that none of the perceived\nbenefits is unique to normalization. Specifically, we propose fixed-update\ninitialization (Fixup), an initialization motivated by solving the exploding\nand vanishing gradient problem at the beginning of training via properly\nrescaling a standard initialization. We find training residual networks with\nFixup to be as stable as training with normalization -- even for networks with\n10,000 layers. Furthermore, with proper regularization, Fixup enables residual\nnetworks without normalization to achieve state-of-the-art performance in image\nclassification and machine translation.","url_abs":"http://arxiv.org/abs/1901.09321v2","url_pdf":"http://arxiv.org/pdf/1901.09321v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/Abhimanyu08/Fixup_Initialization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/AngusG/bn-advex-zhang-fixup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/Zelgunn/CustomKerasLayers","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}},{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/ben-davidson-6/fixup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/bzhangGo/zero","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/hongyi-zhang/Fixup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/yanivbl6/fixup","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/MindCode-4/code-11/tree/main/Fixup-Initialization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/MindCode-4/code-7/tree/main/Fixup-Initialization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"fixup-initialization-residual-learning","repo_url":"https://github.com/MindSpore-scientific-2/code-12/tree/main/Fixup-Initialization","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"fixup-initialization","method_name":"Fixup Initialization"}],"datasets_introduced":[],"methods_introduced":[{"slug":"fixup-initialization","name":"Fixup Initialization","full_name":"Fixup Initialization"}],"results":[{"leaderboard":"/sota/image-classification-on-cifar-10","task":"Image Classification","dataset":"CIFAR-10","model":"WRN + fixup init + mixup + cutout","rank_in_archive_order":74,"of":265,"metrics":{"Percentage correct":"97.7"},"uses_additional_data":false},{"leaderboard":"/sota/image-classification-on-svhn","task":"Image Classification","dataset":"SVHN","model":"WRN + fixup init + mixup + cutout","rank_in_archive_order":8,"of":62,"metrics":{"Percentage error":"1.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1901.09321","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1901.09321"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hongyi-zhang/Fixup","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bzhangGo/zero","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yanivbl6/fixup","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Zelgunn/CustomKerasLayers","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AngusG/bn-advex-zhang-fixup","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Abhimanyu08/Fixup_Initialization","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-11/tree/main/Fixup-Initialization","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindSpore-scientific-2/code-12/tree/main/Fixup-Initialization","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MindCode-4/code-7/tree/main/Fixup-Initialization","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ben-davidson-6/fixup","reach":null}],"summary":{"ran_draft_wrong":1,"ran_honours":1,"unverified":2},"by_repo_kind":{"listed":{"samples":2,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"f380c26e19e9072f","entry":"conv","repo":"Abhimanyu08/Fixup_Initialization","repo_kind":"listed","path":"model_build.py","file_url":"https://github.com/Abhimanyu08/Fixup_Initialization/blob/HEAD/model_build.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f380c26e19e9072f"}},{"code_sha256_prefix":"5e2ed23acb71ee11","entry":"noop","repo":"Abhimanyu08/Fixup_Initialization","repo_kind":"listed","path":"model_build.py","file_url":"https://github.com/Abhimanyu08/Fixup_Initialization/blob/HEAD/model_build.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5e2ed23acb71ee11"}},{"code_sha256_prefix":"9b8289076669fe4f","entry":"accuracy","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"9b8289076669fe4f"}},{"code_sha256_prefix":"d07d18ed4137d374","entry":"validate","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"d07d18ed4137d374"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}