{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/exponential-expressivity-in-deep-neural","title":"Exponential expressivity in deep neural networks through transient chaos","arxiv_id":"1606.05340","date":"2016-06-16","proceeding":"NeurIPS 2016 12","authors":["Ben Poole","Subhaneil Lahiri","Maithra Raghu","Jascha Sohl-Dickstein","Surya Ganguli"],"abstract":"We combine Riemannian geometry with the mean field theory of high dimensional\nchaos to study the nature of signal propagation in generic, deep neural\nnetworks with random weights. Our results reveal an order-to-chaos expressivity\nphase transition, with networks in the chaotic phase computing nonlinear\nfunctions whose global curvature grows exponentially with depth but not width.\nWe prove this generic class of deep random functions cannot be efficiently\ncomputed by any shallow network, going beyond prior work restricted to the\nanalysis of single functions. Moreover, we formalize and quantitatively\ndemonstrate the long conjectured idea that deep networks can disentangle highly\ncurved manifolds in input space into flat manifolds in hidden space. Our\ntheoretical analysis of the expressive power of deep networks broadly applies\nto arbitrary nonlinearities, and provides a quantitative underpinning for\npreviously abstract notions about the geometry of deep functions.","url_abs":"http://arxiv.org/abs/1606.05340v2","url_pdf":"http://arxiv.org/pdf/1606.05340v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"exponential-expressivity-in-deep-neural","repo_url":"https://github.com/ganguli-lab/deepchaos","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1606.05340","atlas_url":"https://app.syntology.ai/?focus=1606.05340","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}