{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/step-size-matters-in-deep-learning","title":"Step Size Matters in Deep Learning","arxiv_id":"1805.08890","date":"2018-05-22","proceeding":"NeurIPS 2018 12","authors":["Kamil Nar","S. Shankar Sastry"],"abstract":"Training a neural network with the gradient descent algorithm gives rise to a\ndiscrete-time nonlinear dynamical system. Consequently, behaviors that are\ntypically observed in these systems emerge during training, such as convergence\nto an orbit but not to a fixed point or dependence of convergence on the\ninitialization. Step size of the algorithm plays a critical role in these\nbehaviors: it determines the subset of the local optima that the algorithm can\nconverge to, and it specifies the magnitude of the oscillations if the\nalgorithm converges to an orbit. To elucidate the effects of the step size on\ntraining of neural networks, we study the gradient descent algorithm as a\ndiscrete-time dynamical system, and by analyzing the Lyapunov stability of\ndifferent solutions, we show the relationship between the step size of the\nalgorithm and the solutions that can be obtained with this algorithm. The\nresults provide an explanation for several phenomena observed in practice,\nincluding the deterioration in the training error with increased depth, the\nhardness of estimating linear mappings with large singular values, and the\ndistinct performance of deep residual networks.","url_abs":"http://arxiv.org/abs/1805.08890v2","url_pdf":"http://arxiv.org/pdf/1805.08890v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"step-size-matters-in-deep-learning","repo_url":"https://github.com/nar-k/NIPS-2018","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}},{"paper_slug":"step-size-matters-in-deep-learning","repo_url":"https://github.com/nar-k/NeurIPS-2018","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1805.08890","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}