{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/accelerating-natural-gradient-with-higher","title":"Accelerating Natural Gradient with Higher-Order Invariance","arxiv_id":"1803.01273","date":"2018-03-04","proceeding":"ICML 2018 7","authors":["Yang Song","Jiaming Song","Stefano Ermon"],"abstract":"An appealing property of the natural gradient is that it is invariant to\narbitrary differentiable reparameterizations of the model. However, this\ninvariance property requires infinitesimal steps and is lost in practical\nimplementations with small but finite step sizes. In this paper, we study\ninvariance properties from a combined perspective of Riemannian geometry and\nnumerical differential equation solving. We define the order of invariance of a\nnumerical method to be its convergence order to an invariant solution. We\npropose to use higher-order integrators and geodesic corrections to obtain more\ninvariant optimization trajectories. We prove the numerical convergence\nproperties of geodesic corrected updates and show that they can be as\ncomputationally efficient as plain natural gradient. Experimentally, we\ndemonstrate that invariance leads to faster optimization and our techniques\nimprove on traditional natural gradient in deep neural network training and\nnatural policy gradient for reinforcement learning.","url_abs":"http://arxiv.org/abs/1803.01273v2","url_pdf":"http://arxiv.org/pdf/1803.01273v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"accelerating-natural-gradient-with-higher","repo_url":"https://github.com/ermongroup/higher_order_invariance","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"accelerating-natural-gradient-with-higher","repo_url":"https://github.com/ferrine/torch_anatgrad","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1803.01273","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1803.01273"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ermongroup/higher_order_invariance","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ferrine/torch_anatgrad","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":2},"by_repo_kind":{"listed":{"samples":2,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6aff48ec0d1463e1","entry":"compute_cov_a","repo":"ferrine/torch_anatgrad","repo_kind":"listed","path":"torch_anatgrad/kfac.py","file_url":"https://github.com/ferrine/torch_anatgrad/blob/HEAD/torch_anatgrad/kfac.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6aff48ec0d1463e1"}},{"code_sha256_prefix":"d9fd61d47e236b09","entry":"compute_cov_g","repo":"ferrine/torch_anatgrad","repo_kind":"listed","path":"torch_anatgrad/kfac.py","file_url":"https://github.com/ferrine/torch_anatgrad/blob/HEAD/torch_anatgrad/kfac.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d9fd61d47e236b09"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}