{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-stabilizing-batch-statistics-in-1","title":"Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization","arxiv_id":"2001.06838","date":"2020-01-19","proceeding":"ICLR 2020 1","authors":["Junjie Yan","Ruosi Wan","Xiangyu Zhang","Wei zhang","Yichen Wei","Jian Sun"],"abstract":"Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field. But its performance can awfully degrade with insufficient batch size. This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption. Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption. In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before. The extra batch statistics associated with gradients also can severely affect the training of deep neural network. Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN). MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure. We prove the benefits of MABN by both theoretical analysis and experiments. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO. The code has been released in https://github.com/megvii-model/MABN.","url_abs":"https://arxiv.org/abs/2001.06838v2","url_pdf":"https://arxiv.org/pdf/2001.06838v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-stabilizing-batch-statistics-in-1","repo_url":"https://github.com/megvii-model/MABN","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"batch-normalization","method_name":"Batch Normalization"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2001.06838","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2001.06838"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/megvii-model/MABN","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"469bf48752905889","entry":"accuracy","repo":"megvii-model/MABN","repo_kind":"official","path":"cls/utils.py","file_url":"https://github.com/megvii-model/MABN/blob/HEAD/cls/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"469bf48752905889"}},{"code_sha256_prefix":"7a48395822d92761","entry":"conv1x1","repo":"megvii-model/MABN","repo_kind":"official","path":"cls/networks/resnet.py","file_url":"https://github.com/megvii-model/MABN/blob/HEAD/cls/networks/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7a48395822d92761"}},{"code_sha256_prefix":"1e5865e06cfa8e5d","entry":"conv3x3","repo":"megvii-model/MABN","repo_kind":"official","path":"cls/networks/resnet.py","file_url":"https://github.com/megvii-model/MABN/blob/HEAD/cls/networks/resnet.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1e5865e06cfa8e5d"}},{"code_sha256_prefix":"2dd50f3ea73256ab","entry":"get_train_dataloader","repo":"megvii-model/MABN","repo_kind":"official","path":"cls/utils.py","file_url":"https://github.com/megvii-model/MABN/blob/HEAD/cls/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dd50f3ea73256ab"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}