{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/local-minima-in-training-of-neural-networks","title":"Local minima in training of neural networks","arxiv_id":"1611.06310","date":"2016-11-19","proceeding":null,"authors":["Grzegorz Swirszcz","Wojciech Marian Czarnecki","Razvan Pascanu"],"abstract":"There has been a lot of recent interest in trying to characterize the error\nsurface of deep models. This stems from a long standing question. Given that\ndeep networks are highly nonlinear systems optimized by local gradient methods,\nwhy do they not seem to be affected by bad local minima? It is widely believed\nthat training of deep models using gradient methods works so well because the\nerror surface either has no local minima, or if they exist they need to be\nclose in value to the global minimum. It is known that such results hold under\nvery strong assumptions which are not satisfied by real models. In this paper\nwe present examples showing that for such theorem to be true additional\nassumptions on the data, initialization schemes and/or the model classes have\nto be made. We look at the particular case of finite size datasets. We\ndemonstrate that in this scenario one can construct counter-examples (datasets\nor initialization schemes) when the network does become susceptible to bad\nlocal minima over the weight space.","url_abs":"http://arxiv.org/abs/1611.06310v2","url_pdf":"http://arxiv.org/pdf/1611.06310v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"local-minima-in-training-of-neural-networks","repo_url":"https://github.com/jchunn/Ambition","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1611.06310","atlas_url":"https://app.syntology.ai/?focus=1611.06310","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}