{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/fast-approximate-natural-gradient-descent-in-1","title":"Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis","arxiv_id":"1806.03884","date":"2018-06-11","proceeding":null,"authors":["Thomas George","César Laurent","Xavier Bouthillier","Nicolas Ballas","Pascal Vincent"],"abstract":"Optimization algorithms that leverage gradient covariance information, such as variants of natural gradient descent (Amari, 1998), offer the prospect of yielding more effective descent directions. For models with many parameters, the covariance matrix they are based on becomes gigantic, making them inapplicable in their original form. This has motivated research into both simple diagonal approximations and more sophisticated factored approximations such as KFAC (Heskes, 2000; Martens & Grosse, 2015; Grosse & Martens, 2016). In the present work we draw inspiration from both to propose a novel approximation that is provably better than KFAC and amendable to cheap partial updates. It consists in tracking a diagonal variance, not in parameter coordinates, but in a Kronecker-factored eigenbasis, in which the diagonal approximation is likely to be more effective. Experiments show improvements over KFAC in optimization speed for several deep network architectures.","url_abs":"https://arxiv.org/abs/1806.03884v2","url_pdf":"https://arxiv.org/pdf/1806.03884v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"fast-approximate-natural-gradient-descent-in-1","repo_url":"https://github.com/tfjgeorge/nngeometry","is_official":0,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null},{"paper_slug":"fast-approximate-natural-gradient-descent-in-1","repo_url":"https://github.com/DLR-RM/curvature","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"GPL-3.0"}},{"paper_slug":"fast-approximate-natural-gradient-descent-in-1","repo_url":"https://github.com/Thrandis/EKFAC-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fast-approximate-natural-gradient-descent-in-1","repo_url":"https://github.com/aai-institute/pyDVL","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"LGPL-3.0"}},{"paper_slug":"fast-approximate-natural-gradient-descent-in-1","repo_url":"https://github.com/alecwangcq/KFAC-Pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"fast-approximate-natural-gradient-descent-in-1","repo_url":"https://github.com/pomonam/kronfluence","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1806.03884","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1806.03884"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Thrandis/EKFAC-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pomonam/kronfluence","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aai-institute/pyDVL","reach":{"status":"ok","spdx":"LGPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alecwangcq/KFAC-Pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/DLR-RM/curvature","reach":{"status":"ok","spdx":"GPL-3.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tfjgeorge/nngeometry","reach":null}],"summary":{"ran_draft_wrong":1},"by_repo_kind":{"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c95310b1a58f89e9","entry":"grad_wrt_kernel","repo":"Thrandis/EKFAC-pytorch","repo_kind":"listed","path":"ekfac.py","file_url":"https://github.com/Thrandis/EKFAC-pytorch/blob/HEAD/ekfac.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c95310b1a58f89e9"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}