{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/comparing-dynamics-deep-neural-networks-1","title":"Comparing Dynamics: Deep Neural Networks versus Glassy Systems","arxiv_id":"1803.06969","date":"2018-03-19","proceeding":"ICML 2018","authors":["M. Baity-Jesi","L. Sagun","M. Geiger","S. Spigler","G. Ben Arous","C. Cammarota","Y. LeCun","M. Wyart","G. Biroli"],"abstract":"We analyze numerically the training dynamics of deep neural networks (DNN) by\nusing methods developed in statistical physics of glassy systems. The two main\nissues we address are (1) the complexity of the loss landscape and of the\ndynamics within it, and (2) to what extent DNNs share similarities with glassy\nsystems. Our findings, obtained for different architectures and datasets,\nsuggest that during the training process the dynamics slows down because of an\nincreasingly large number of flat directions. At large times, when the loss is\napproaching zero, the system diffuses at the bottom of the landscape. Despite\nsome similarities with the dynamics of mean-field glassy systems, in\nparticular, the absence of barrier crossing, we find distinctive dynamical\nbehaviors in the two cases, showing that the statistical properties of the\ncorresponding loss and energy landscapes are different. In contrast, when the\nnetwork is under-parametrized we observe a typical glassy behavior, thus\nsuggesting the existence of different phases depending on whether the network\nis under-parametrized or over-parametrized.","url_abs":"http://arxiv.org/abs/1803.06969v2","url_pdf":"http://arxiv.org/pdf/1803.06969v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"comparing-dynamics-deep-neural-networks-1","repo_url":"https://github.com/mbaityje/DEEP-GLASS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.06969","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}