{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/critical-initialisation-for-deep-signal","title":"Critical initialisation for deep signal propagation in noisy rectifier neural networks","arxiv_id":"1811.00293","date":"2018-11-01","proceeding":"NeurIPS 2018 12","authors":["Arnu Pretorius","Elan van Biljon","Steve Kroon","Herman Kamper"],"abstract":"Stochastic regularisation is an important weapon in the arsenal of a deep\nlearning practitioner. However, despite recent theoretical advances, our\nunderstanding of how noise influences signal propagation in deep neural\nnetworks remains limited. By extending recent work based on mean field theory,\nwe develop a new framework for signal propagation in stochastic regularised\nneural networks. Our noisy signal propagation theory can incorporate several\ncommon noise distributions, including additive and multiplicative Gaussian\nnoise as well as dropout. We use this framework to investigate initialisation\nstrategies for noisy ReLU networks. We show that no critical initialisation\nstrategy exists using additive noise, with signal propagation exploding\nregardless of the selected noise distribution. For multiplicative noise (e.g.\ndropout), we identify alternative critical initialisation strategies that\ndepend on the second moment of the noise distribution. Simulations and\nexperiments on real-world data confirm that our proposed initialisation is able\nto stably propagate signals in deep networks, while using an initialisation\ndisregarding noise fails to do so. Furthermore, we analyse correlation dynamics\nbetween inputs. Stronger noise regularisation is shown to reduce the depth to\nwhich discriminatory information about the inputs to a noisy ReLU network is\nable to propagate, even when initialised at criticality. We support our\ntheoretical predictions for these trainable depths with simulations, as well as\nwith experiments on MNIST and CIFAR-10","url_abs":"http://arxiv.org/abs/1811.00293v2","url_pdf":"http://arxiv.org/pdf/1811.00293v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"critical-initialisation-for-deep-signal","repo_url":"https://github.com/ElanVB/noisy_signal_prop","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[],"methods":[{"method_slug":"relu","method_name":"ReLU"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}