{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-power-of-over-parametrization-in-1","title":"On the Power of Over-parametrization in Neural Networks with Quadratic Activation","arxiv_id":"1803.01206","date":"2018-03-03","proceeding":"ICML 2018","authors":["Simon S. Du","Jason D. Lee"],"abstract":"We provide new theoretical insights on why over-parametrization is effective\nin learning neural networks. For a $k$ hidden node shallow network with\nquadratic activation and $n$ training data points, we show as long as $ k \\ge\n\\sqrt{2n}$, over-parametrization enables local search algorithms to find a\n\\emph{globally} optimal solution for general smooth and convex loss functions.\nFurther, despite that the number of parameters may exceed the sample size,\nusing theory of Rademacher complexity, we show with weight decay, the solution\nalso generalizes well if the data is sampled from a regular distribution such\nas Gaussian. To prove when $k\\ge \\sqrt{2n}$, the loss function has benign\nlandscape properties, we adopt an idea from smoothed analysis, which may have\nother applications in studying loss surfaces of neural networks.","url_abs":"http://arxiv.org/abs/1803.01206v2","url_pdf":"http://arxiv.org/pdf/1803.01206v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-power-of-over-parametrization-in-1","repo_url":"https://github.com/Clumsyndicate/One_layer_analysis_network","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1803.01206","atlas_url":"https://app.syntology.ai/?focus=1803.01206","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}