{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/new-insights-and-perspectives-on-the-natural","title":"New insights and perspectives on the natural gradient method","arxiv_id":"1412.1193","date":"2014-12-03","proceeding":null,"authors":["James Martens"],"abstract":"Natural gradient descent is an optimization method traditionally motivated\nfrom the perspective of information geometry, and works well for many\napplications as an alternative to stochastic gradient descent. In this paper we\ncritically analyze this method and its properties, and show how it can be\nviewed as a type of approximate 2nd-order optimization method, where the Fisher\ninformation matrix can be viewed as an approximation of the Hessian. This\nperspective turns out to have significant implications for how to design a\npractical and robust version of the method. Additionally, we make the following\ncontributions to the understanding of natural gradient and 2nd-order methods: a\nthorough analysis of the convergence speed of stochastic natural gradient\ndescent (and more general stochastic 2nd-order methods) as applied to convex\nquadratics, a critical examination of the oft-used \"empirical\" approximation of\nthe Fisher matrix, and an analysis of the (approximate) parameterization\ninvariance property possessed by natural gradient methods, which we show still\nholds for certain choices of the curvature matrix other than the Fisher, but\nnotably not the Hessian.","url_abs":"http://arxiv.org/abs/1412.1193v9","url_pdf":"http://arxiv.org/pdf/1412.1193v9.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"new-insights-and-perspectives-on-the-natural","repo_url":"https://github.com/lantonov/Optimisation","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"new-insights-and-perspectives-on-the-natural","repo_url":"https://github.com/lixilinx/psgd_torch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1412.1193","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}