Second-order methods

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

No benchmark for this task in the archive.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

No dataset record in the archive lists this task.

Subtasks archive 2025-07-28

No subtask under this task in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 52 papers with code (181 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning
  1 Jun 2020 4 repositories listed Syntology ran 1 of 11 samples · 10 unverified
  We introduce ADAHESSIAN, a second order stochastic optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates of the HESSIAN.
• Second-Order Stochastic Optimization for Machine Learning in Linear Time
  12 Feb 2016 4 repositories listed
  First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity.
• Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective
  5 Feb 2024 2 repositories listed Syntology ran 9 of 10 samples · 1 unverified · 9 pointer-only (licence)
  Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers.
• Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC
  9 Dec 2023 2 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Second-order methods such as KFAC can be useful for neural net training.
• Near out-of-distribution detection for low-resolution radar micro-Doppler signatures
  12 May 2022 2 repositories listed
  We emphasize the relevance of OODD and its specific supervision requirements for the detection of a multimodal, diverse targets class among other similar radar targets and clutter in real-life critical systems.
• M-FAC: Efficient Matrix-Free Approximations of Second-Order Information
  7 Jul 2021 2 repositories listed Syntology ran 7 of 14 samples · 7 unverified
  We propose two new algorithms as part of a framework called M-FAC: the first algorithm is tailored towards network compression and can compute the IHVP for dimension d, if the Hessian is given as a sum of m rank-one…
• On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNs
  3 Jun 2020 2 repositories listed
  This enables researchers to further study and improve this promising optimization technique and hopefully reconsider stochastic second-order methods as competitive optimization techniques for training DNNs; we also hope…
• Low Rank Saddle Free Newton: A Scalable Method for Stochastic Nonconvex Optimization
  7 Feb 2020 2 repositories listed
  In this work we motivate the extension of Newton methods to the SA regime, and argue for the use of the scalable low rank saddle free Newton (LRSFN) method, which avoids forming the Hessian in favor of making a low rank…
• Newtonian Monte Carlo: single-site MCMC meets second-order gradient methods
  15 Jan 2020 2 repositories listed
  NMC is similar to the Newton-Raphson update in optimization where the second order gradient is used to automatically scale the step size in each dimension.
• NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation
  10 Jun 2025 1 repository listed
  Adaptive gradient methods are computationally efficient and converge quickly, but they often suffer from poor generalization.
• Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis
  26 Apr 2025 1 repository listed
  AdaFisher aims to bridge the gap between the improved convergence and generalization of second-order methods and the computational efficiency needed for training DNNs.
• SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
  25 Feb 2025 1 repository listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches.
• Online Covariance Matrix Estimation in Sketched Newton Methods
  10 Feb 2025 1 repository listed
  Given the ubiquity of streaming data, online algorithms have been widely used for parameter estimation, with second-order methods particularly standing out for their efficiency and robustness.
• Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization
  12 Nov 2024 1 repository listed Syntology ran 0 of 3 samples · 3 unverified
  We exploit this novel setting to study the training and generalization properties of the GN optimizer.
• Alternating Iteratively Reweighted ℓ₁ and Subspace Newton Algorithms for Nonconvex Sparse Optimization
  24 Jul 2024 1 repository listed
  This paper presents a novel hybrid algorithm for minimizing the sum of a continuously differentiable loss function and a nonsmooth, possibly nonconvex, sparse regularization function.
• Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning
  5 Jun 2024 1 repository listed
  Second-order information is valuable for many applications but challenging to compute.
• A Gauss-Newton Approach for Min-Max Optimization in Generative Adversarial Networks
  10 Apr 2024 1 repository listed
  It modifies the Gauss-Newton method to approximate the min-max Hessian and uses the Sherman-Morrison inversion formula to calculate the inverse.
• SGD with Partial Hessian for Deep Neural Networks Optimization
  5 Mar 2024 1 repository listed
  Compared with first-order optimizers, it adopts a certain amount of information from the Hessian matrix to assist optimization, while compared with the existing second-order optimizers, it keeps the good generalization…
• Adapting Newton's Method to Neural Networks through a Summary of Higher-Order Derivatives
  6 Dec 2023 1 repository listed
  When training large models, such as neural networks, the full derivatives of order 2 and beyond are usually inaccessible, due to their computational cost.
• AdaSub: Stochastic Optimization Using Second-Order Information in Low-Dimensional Subspaces
  30 Oct 2023 1 repository listed
  We introduce AdaSub, a stochastic optimization algorithm that computes a search direction based on second-order information in a low-dimensional subspace that is defined adaptively based on available current and past…
• Studying K-FAC Heuristics by Viewing Adam through a Second-Order Lens
  23 Oct 2023 1 repository listed Syntology ran 7 of 16 samples · 9 unverified · 16 pointer-only (licence)
  Research into optimisation for deep learning is characterised by a tension between the computational efficiency of first-order, gradient-based methods (such as SGD and Adam) and the theoretical efficiency of…
• Error Feedback Can Accurately Compress Preconditioners
  9 Jun 2023 1 repository listed Syntology ran 2 of 3 samples · 1 unverified
  Experiments on deep neural networks show that this approach can compress full-matrix preconditioners to up to 99\% sparsity without accuracy loss, effectively removing the memory overhead of full-matrix preconditioners…
• MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates
  2 Jun 2023 1 repository listed Syntology ran 9 of 10 samples · 1 unverified · 10 pointer-only (licence)
  This work proposes a Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 updates, called MKOR, that improves the training time and convergence properties of deep neural networks (DNNs).
• Sharpened Lazy Incremental Quasi-Newton Method
  26 May 2023 1 repository listed
  Specifically, they exhibit a superlinear rate with O(d²) cost in contrast to the linear rate of first-order methods with O(d) cost and the quadratic rate of second-order methods with O(d³) cost.
• ISAAC Newton: Input-based Approximate Curvature for Newton's Method
  1 May 2023 1 repository listed Syntology ran 0 of 1 samples · 1 unverified
  We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size…
• Automatic Gradient Descent: Deep Learning without Hyperparameters
  11 Apr 2023 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Automatic gradient descent trains both fully-connected and convolutional networks out-of-the-box and at ImageNet scale.
• FOSI: Hybrid First and Second Order Optimization
  16 Feb 2023 1 repository listed Syntology ran 0 of 1 samples · 1 unverified
  Popular machine learning approaches forgo second-order information due to the difficulty of computing curvature in high dimensions.
• Differentially Private Image Classification from Features
  24 Nov 2022 1 repository listed
  We find that linear regression is much more effective than logistic regression from both privacy and computational aspects, especially at stricter epsilon values (ϵ< 1).
• The Hypervolume Indicator Hessian Matrix: Analytical Expression, Computational Time Complexity, and Sparsity
  8 Nov 2022 1 repository listed
  Also, for the general m-dimensional case, a compact recursive analytical expression is established, and its algorithmic implementation is discussed.
• Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batches
  6 Jun 2022 1 repository listed
  Yet, comparatively little is known about the benefits of using variance reduction to accelerate popular stochastic second-order methods such as Subsampled Newton.

Syntology lines on 12 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.