Methods › General › Optimization › Natural Gradient Descent
Natural Gradient Descent
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Natural Gradient Descent is an approximate second-order optimisation method. It has an interpretation as optimizing over a Riemannian manifold using an intrinsic distance metric, which implies the updates are invariant to transformations such as whitening. By using the positive semi-definite (PSD) Gauss-Newton matrix to approximate the (possibly negative definite) Hessian, NGD can often work better than exact second-order methods.
Given the gradient of z, g = (δf(z))/δz, NGD computes the update as:
Δz = αF⁻¹g
where the Fisher information matrix F is defined as:
F = 𝔼_(p(t|z))[∇lnp(t|z)∇lnp(t|z)ᵀ]
The log-likelihood function lnp(t|z) typically corresponds to commonly used error functions such as the cross entropy loss.
Source: LOGAN
Image: Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
In Syntology name not resolved (unknown_name). The lookup matched no method, paper or venue by this name; that asserts nothing about whether code exists.
Near matches the lookup lists, each a different name: natural Riemannian gradient descent; Natural Hypergradient Descent (NHGD); GAUSS-NEWTON NATURAL GRADIENT DESCENT; natural gradient descent with momentum; Tractable Structured Natural-Gradient Descent; Zeroth-Order Natural Gradient Descent (ZO-NGD); Energy Manifold Natural Gradient Descent (EMNGD); Natural Gradient Descent for Online Continual Learning.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 68 · with a code link: 21 · where Syntology ran a sample: 6 (5 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (6 of 68 tagged: 5 with a run with no instrument failure, 1 where every run was a failure of Syntology's instrument)
Page 1 of 1: papers 1 to 68 of 68, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization 17 May 2025 · 0 repositories · arXiv:2505.12149
-
Convergence Properties of Natural Gradient Descent for Minimizing KL Divergence 27 Apr 2025 · 0 repositories · arXiv:2504.19259
-
A mean teacher algorithm for unlearning of language models 18 Apr 2025 · 1 repository · arXiv:2504.13388
-
Natural Gradient Descent for Control 8 Mar 2025 · 0 repositories · arXiv:2503.06070
-
Guiding Time-Varying Generative Models with Natural Gradients on Exponential Family Manifold 11 Feb 2025 · 0 repositories · arXiv:2502.07650
-
High-accuracy sampling from constrained spaces with the Metropolis-adjusted Preconditioned Langevin Algorithm 24 Dec 2024 · 1 repository · arXiv:2412.18701Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)
-
Reconstructing Deep Neural Networks: Unleashing the Optimization Potential of Natural Gradient Descent 10 Dec 2024 · 1 repository · arXiv:2412.07441
-
Natural gradient and parameter estimation for quantum Boltzmann machines 31 Oct 2024 · 0 repositories · arXiv:2410.24058
-
Brain-like variational inference 25 Oct 2024 · 0 repositories · arXiv:2410.19315Syntology 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified (of 6 harvested samples)
-
Score-Based Variational Inference for Inverse Problems 8 Oct 2024 · 0 repositories · arXiv:2410.05646
-
Is All Learning (Natural) Gradient Descent? 24 Sep 2024 · 0 repositories · arXiv:2409.16422
-
NGD converges to less degenerate solutions than SGD 7 Sep 2024 · 1 repository · arXiv:2409.04913
-
Decentralised Variational Inference Frameworks for Multi-object Tracking on Sensor Networks: Additional Notes 24 Aug 2024 · 0 repositories · arXiv:2408.13689
-
Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks 1 Aug 2024 · 0 repositories · arXiv:2408.00573
-
Quantum Natural Stochastic Pairwise Coordinate Descent 18 Jul 2024 · 1 repository · arXiv:2407.13858
-
Correlations Are Ruining Your Gradient Descent 15 Jul 2024 · 1 repository · arXiv:2407.10780
-
Faster Machine Unlearning via Natural Gradient Descent 11 Jul 2024 · 0 repositories · arXiv:2407.08169
-
An Improved Empirical Fisher Approximation for Natural Gradient Descent 10 Jun 2024 · 0 repositories · arXiv:2406.06420
-
Bayesian Online Natural Gradient (BONG) 30 May 2024 · 1 repository · arXiv:2405.19681Syntology official (archive's flag): 9 ran · 9 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 9 harvested samples)
-
Thermodynamic Natural Gradient Descent 22 May 2024 · 0 repositories · arXiv:2405.13817
-
FAdam: Adam is a natural gradient optimizer using diagonal empirical Fisher information 21 May 2024 · 1 repository · arXiv:2405.12807
-
Sequential-in-time training of nonlinear parametrizations for solving time-dependent partial differential equations 1 Apr 2024 · 0 repositories · arXiv:2404.01145
-
Inverse-Free Fast Natural Gradient Descent Method for Deep Learning 6 Mar 2024 · 0 repositories · arXiv:2403.03473
-
SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix 5 Mar 2024 · 0 repositories · arXiv:2403.02833
-
A Kaczmarz-inspired approach to accelerate the optimization of neural network wavefunctions 18 Jan 2024 · 1 repository · arXiv:2401.10190
-
Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC 9 Dec 2023 · 2 repositories · arXiv:2312.05705Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Efficient Numerical Algorithm for Large-Scale Damped Natural Gradient Descent 26 Oct 2023 · 0 repositories · arXiv:2310.17556
-
Modify Training Directions in Function Space to Reduce Generalization Error 25 Jul 2023 · 0 repositories · arXiv:2307.13290
-
Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networks 31 Mar 2023 · 1 repository · arXiv:2303.18083
-
Decentralized Riemannian natural gradient methods with Kronecker-product approximations 16 Mar 2023 · 0 repositories · arXiv:2303.09611
-
Optimizing Quantum Federated Learning Based on Federated Quantum Natural Gradient Descent 27 Feb 2023 · 0 repositories · arXiv:2303.08116
-
Achieving High Accuracy with PINNs via Energy Natural Gradients 25 Feb 2023 · 1 repository · arXiv:2302.13163
-
Image Reconstruction via Deep Image Prior Subspaces 20 Feb 2023 · 1 repository · arXiv:2302.10279
-
Auditing Algorithmic Fairness in Machine Learning for Health with Severity-Based LOGAN 16 Nov 2022 · 0 repositories · arXiv:2211.08742
-
Component-Wise Natural Gradient Descent -- An Efficient Neural Network Optimization 11 Oct 2022 · 0 repositories · arXiv:2210.05268
-
Conjugate Natural Selection 29 Aug 2022 · 0 repositories · arXiv:2208.13898
-
Federated Quantum Natural Gradient Descent for Quantum Federated Learning 15 Aug 2022 · 0 repositories · arXiv:2209.00564
-
QSAN: A Near-term Achievable Quantum Self-Attention Network 14 Jul 2022 · 0 repositories · arXiv:2207.07563
-
The Galactic 3D large-scale dust distribution via Gaussian process regression on spherical coordinates 25 Apr 2022 · 0 repositories · arXiv:2204.11715
-
Amortized Proximal Optimization 28 Feb 2022 · 0 repositories · arXiv:2203.00089
-
Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems 13 Feb 2022 · 0 repositories · arXiv:2202.06236
-
Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decomposition 25 Jan 2022 · 1 repository · arXiv:2201.10285
-
Depth Without the Magic: Inductive Bias of Natural Gradient Descent 22 Nov 2021 · 0 repositories · arXiv:2111.11542
-
PredProp: Bidirectional Stochastic Optimization with Precision Weighted Predictive Coding 16 Nov 2021 · 0 repositories · arXiv:2111.08792
-
Predictive coding, precision and natural gradients 12 Nov 2021 · 0 repositories · arXiv:2111.06942
-
Variational Inference at Glacier Scale 16 Aug 2021 · 0 repositories · arXiv:2108.07263
-
Sinogram Denoise Based on Generative Adversarial Networks 9 Aug 2021 · 0 repositories · arXiv:2108.03903
-
Rayleigh-Gauss-Newton optimization with enhanced sampling for variational Monte Carlo 19 Jun 2021 · 0 repositories · arXiv:2106.10558
-
Direct Reconstruction of Linear Parametric Images from Dynamic PET Using Nonlocal Deep Image Prior 18 Jun 2021 · 0 repositories · arXiv:2106.10359
-
TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion 7 Jun 2021 · 1 repository · arXiv:2106.03947
-
THOR, Trace-based Hardware-adaptive layer-ORiented Natural Gradient Descent Computation 18 May 2021 · 0 repositories
-
Noise-Resilient Quantum Machine Learning for Stability Assessment of Power Systems 10 Apr 2021 · 0 repositories · arXiv:2104.04855
-
Moment-Based Variational Inference for Stochastic Differential Equations 1 Mar 2021 · 0 repositories · arXiv:2103.00988
-
Optimizing Quantized Neural Networks with Natural Gradient 1 Jan 2021 · 0 repositories
-
Two-Level K-FAC Preconditioning for Deep Learning 1 Nov 2020 · 1 repository · arXiv:2011.00573
-
LOGAN: Local Group Bias Detection by Clustering 6 Oct 2020 · 1 repository · arXiv:2010.02867
-
Allpass Feedback Delay Networks 14 Jul 2020 · 0 repositories · arXiv:2007.07337
-
When Does Preconditioning Help or Hurt Generalization? 18 Jun 2020 · 0 repositories · arXiv:2006.10732
-
Mirrorless Mirror Descent: A Natural Derivation of Mirror Descent 2 Apr 2020 · 0 repositories · arXiv:2004.01025
-
Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descent 18 Feb 2020 · 1 repository · arXiv:2002.07891Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Scalable and Practical Natural Gradient for Large-Scale Deep Learning 13 Feb 2020 · 1 repository · arXiv:2002.06015Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
Multivariate Gaussian Variational Inference by Natural Gradient Descent 27 Jan 2020 · 0 repositories · arXiv:2001.10025
-
Dual Stochastic Natural Gradient Descent and convergence of interior half-space gradient approximations 19 Jan 2020 · 0 repositories · arXiv:2001.06744
-
Implicit Regularization and Momentum Algorithms in Nonlinearly Parameterized Adaptive Control and Prediction 31 Dec 2019 · 0 repositories · arXiv:1912.13154
-
LOGAN: Latent Optimisation for Generative Adversarial Networks 2 Dec 2019 · 1 repository · arXiv:1912.00953Syntology 5 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 1 violated, 1 with no contract checked; 3 where Syntology's instrument failed) · 3 unverified (of 8 harvested samples) · 2 pointer-only (licence)
-
Fast Convergence of Natural Gradient Descent for Over-Parameterized Neural Networks 1 Dec 2019 · 0 repositories
-
Quantum Natural Gradient 4 Sep 2019 · 2 repositories · arXiv:1909.02108
-
Hopfield Neural Network Flow: A Geometric Viewpoint 4 Aug 2019 · 0 repositories · arXiv:1908.01270
Tasks archive 2025-07-28
20 shown of 51 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections