{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/diagonal-rescaling-for-neural-networks","title":"Diagonal Rescaling For Neural Networks","arxiv_id":"1705.09319","date":"2017-05-25","proceeding":null,"authors":["Jean Lafond","Nicolas Vasilache","Léon Bottou"],"abstract":"We define a second-order neural network stochastic gradient training\nalgorithm whose block-diagonal structure effectively amounts to normalizing the\nunit activations. Investigating why this algorithm lacks in robustness then\nreveals two interesting insights. The first insight suggests a new way to scale\nthe stepsizes, clarifying popular algorithms such as RMSProp as well as old\nneural network tricks such as fanin stepsize scaling. The second insight\nstresses the practical importance of dealing with fast changes of the curvature\nof the cost.","url_abs":"http://arxiv.org/abs/1705.09319v1","url_pdf":"http://arxiv.org/pdf/1705.09319v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"diagonal-rescaling-for-neural-networks","repo_url":"https://github.com/Thrandis/EKFAC-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"rmsprop","method_name":"RMSProp"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1705.09319","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}