{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/topology-and-geometry-of-half-rectified","title":"Topology and Geometry of Half-Rectified Network Optimization","arxiv_id":"1611.01540","date":"2016-11-04","proceeding":null,"authors":["C. Daniel Freeman","Joan Bruna"],"abstract":"The loss surface of deep neural networks has recently attracted interest in\nthe optimization and machine learning communities as a prime example of\nhigh-dimensional non-convex problem. Some insights were recently gained using\nspin glass models and mean-field approximations, but at the expense of strongly\nsimplifying the nonlinear nature of the model.\n  In this work, we do not make any such assumption and study conditions on the\ndata distribution and model architecture that prevent the existence of bad\nlocal minima. Our theoretical work quantifies and formalizes two important\n\\emph{folklore} facts: (i) the landscape of deep linear networks has a\nradically different topology from that of deep half-rectified ones, and (ii)\nthat the energy landscape in the non-linear case is fundamentally controlled by\nthe interplay between the smoothness of the data distribution and model\nover-parametrization. Our main theoretical contribution is to prove that\nhalf-rectified single layer networks are asymptotically connected, and we\nprovide explicit bounds that reveal the aforementioned interplay.\n  The conditioning of gradient descent is the next challenge we address. We\nstudy this question through the geometry of the level sets, and we introduce an\nalgorithm to efficiently estimate the regularity of such sets on large-scale\nnetworks. Our empirical results show that these level sets remain connected\nthroughout all the learning phase, suggesting a near convex behavior, but they\nbecome exponentially more curvy as the energy level decays, in accordance to\nwhat is observed in practice with very low curvature attractors.","url_abs":"http://arxiv.org/abs/1611.01540v4","url_pdf":"http://arxiv.org/pdf/1611.01540v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"topology-and-geometry-of-half-rectified","repo_url":"https://github.com/danielfreeman11/convex-nets","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"unanswered"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1611.01540","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}