{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/deep-information-propagation","title":"Deep Information Propagation","arxiv_id":"1611.01232","date":"2016-11-04","proceeding":null,"authors":["Samuel S. Schoenholz","Justin Gilmer","Surya Ganguli","Jascha Sohl-Dickstein"],"abstract":"We study the behavior of untrained neural networks whose weights and biases\nare randomly distributed using mean field theory. We show the existence of\ndepth scales that naturally limit the maximum depth of signal propagation\nthrough these random networks. Our main practical result is to show that random\nnetworks may be trained precisely when information can travel through them.\nThus, the depth scales that we identify provide bounds on how deep a network\nmay be trained for a specific choice of hyperparameters. As a corollary to\nthis, we argue that in networks at the edge of chaos, one of these depth scales\ndiverges. Thus arbitrarily deep networks may be trained only sufficiently close\nto criticality. We show that the presence of dropout destroys the\norder-to-chaos critical point and therefore strongly limits the maximum\ntrainable depth for random networks. Finally, we develop a mean field theory\nfor backpropagation and we show that the ordered and chaotic phases correspond\nto regions of vanishing and exploding gradient respectively.","url_abs":"http://arxiv.org/abs/1611.01232v2","url_pdf":"http://arxiv.org/pdf/1611.01232v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"deep-information-propagation","repo_url":"https://github.com/ghliu/mean-field-fcdnn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"dropout","method_name":"Dropout"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1611.01232","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}