Stochastic Optimization

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

12 leaderboard tables shown for this task, 12 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 12 until expanded.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

11 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

2 subtasks in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 337 papers with code (1,387 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Adam: A Method for Stochastic Optimization
  22 Dec 2014 87 repositories listed Syntology ran 18 of 37 samples · 19 unverified · 8 pointer-only (licence)
  We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
  8 Jun 2017 73 repositories listed Syntology ran 3 of 7 samples · 4 unverified · 7 pointer-only (licence)
  To achieve this result, we adopt a hyper-parameter-free linear scaling rule for adjusting learning rates as a function of minibatch size and develop a new warmup scheme that overcomes optimization challenges early in…
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
  1 Apr 2019 32 repositories listed Syntology ran 1 of 11 samples · 10 unverified · 9 pointer-only (licence)
  In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.
• SGDR: Stochastic Gradient Descent with Warm Restarts
  13 Aug 2016 24 repositories listed Syntology ran 10 of 17 samples · 7 unverified · 7 pointer-only (licence)
  Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions.
• On the Variance of the Adaptive Learning Rate and Beyond
  8 Aug 2019 21 repositories listed Syntology ran 4 of 13 samples · 9 unverified
  The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam.
• Lookahead Optimizer: k steps forward, 1 step back
  19 Jul 2019 19 repositories listed
  The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms.
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
  19 Mar 2015 19 repositories listed Syntology ran 41 of 82 samples · 41 unverified · 7 pointer-only (licence)
  This is because the cost of storing and inverting K-FAC's approximation to the curvature matrix does not depend on the amount of data used to estimate it, which is a feature typically associated only with diagonal or…
• Averaging Weights Leads to Wider Optima and Better Generalization
  14 Mar 2018 17 repositories listed Syntology ran 5 of 9 samples · 4 unverified
  Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence.
• Deep learning with Elastic Averaging SGD
  20 Dec 2014 10 repositories listed Syntology ran 0 of 12 samples · 12 unverified
  We empirically demonstrate that in the deep learning setting, due to the existence of many local optima, allowing more exploration can lead to the improved performance.
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
  20 Nov 2019 8 repositories listed Syntology ran 9 of 10 samples · 1 unverified · 2 pointer-only (licence)
  Distributionally robust optimization (DRO) allows us to learn models that instead minimize the worst-case training loss over a set of pre-defined groups.
• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
  23 May 2023 7 repositories listed Syntology ran 6 of 19 samples · 13 unverified
  Given the massive cost of language model pre-training, a non-trivial improvement of the optimization algorithm would lead to a material reduction on the time and cost of training.
• Agnostic Federated Learning
  1 Feb 2019 7 repositories listed Syntology ran 3 of 10 samples · 7 unverified
  A key learning scenario in large-scale applications is that of federated learning, where a centralized model is trained based on data originating from a large number of clients.
• Training Deep Networks without Learning Rates Through Coin Betting
  22 May 2017 6 repositories listed
  Instead, we reduce the optimization process to a game of betting on a coin and propose a learning-rate-free optimal algorithm for this scenario.
• Variational Inference: A Review for Statisticians
  4 Jan 2016 6 repositories listed
  One of the core problems of modern statistics is to approximate difficult-to-compute probability densities.
• Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
  26 Jan 2021 5 repositories listed
  We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods.
• Learning concise representations for regression by evolving networks of trees
  3 Jul 2018 5 repositories listed
  We propose and study a method for learning interpretable representations for the task of regression.
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
  11 Apr 2018 5 repositories listed Syntology ran 0 of 4 samples · 4 unverified
  In several recently proposed stochastic optimization methods (e.
• Eve: A Gradient Based Optimization Method with Locally and Globally Adaptive Learning Rates
  4 Nov 2016 5 repositories listed
  Adaptive gradient methods for stochastic optimization adjust the learning rate for each parameter locally.
• Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification
  6 Dec 2020 4 repositories listed
  Our studies demonstrate that the proposed DAM method improves the performance of optimizing cross-entropy loss by a large margin, and also achieves better performance than optimizing the existing AUC square loss on…
• Personalized Federated Learning with Moreau Envelopes
  16 Jun 2020 4 repositories listed
  Federated learning (FL) is a decentralized and privacy-preserving machine learning technique in which a group of clients collaborate with a server to learn a global model without sharing clients' data.
• ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning
  1 Jun 2020 4 repositories listed Syntology ran 1 of 11 samples · 10 unverified
  We introduce ADAHESSIAN, a second order stochastic optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates of the HESSIAN.
• Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation
  29 Oct 2019 4 repositories listed
  There is an increasing interest in a fast-growing machine learning technique called Federated Learning, in which the model training is distributed over mobile user equipments (UEs), exploiting UEs' local computation and…
• Kronecker Determinantal Point Processes
  26 May 2016 4 repositories listed
  Determinantal Point Processes (DPPs) are probabilistic models over all subsets a ground set of N items.
• Revisiting Distributed Synchronous SGD
  4 Apr 2016 4 repositories listed
  Distributed training of deep learning models on large-scale training data is typically conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from…
• Second-Order Stochastic Optimization for Machine Learning in Linear Time
  12 Feb 2016 4 repositories listed
  First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity.
• A Kronecker-factored approximate Fisher matrix for convolution layers
  3 Feb 2016 4 repositories listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Second-order optimization methods such as natural gradient descent have the potential to speed up training of neural networks by correcting for the curvature of the loss function.
• A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale
  12 Sep 2023 3 repositories listed Syntology ran 0 of 2 samples · 2 unverified · 2 pointer-only (licence)
  It constructs a block-diagonal preconditioner where each block consists of a coarse Kronecker product approximation to full-matrix AdaGrad for each parameter of the neural network.
• Benchopt: Reproducible, efficient and collaborative optimization benchmarks
  27 Jun 2022 3 repositories listed Syntology ran 0 of 10 samples · 10 unverified
  Numerical validation is at the core of machine learning research as it allows to assess the actual impact of new methods, and to confirm the agreement between theory and practice.
• Differentiable Quality Diversity
  7 Jun 2021 3 repositories listed Syntology ran 4 of 6 samples · 2 unverified · 5 pointer-only (licence)
  Quality diversity (QD) is a growing branch of stochastic optimization research that studies the problem of generating an archive of solutions that maximize a given objective function but are also diverse with respect to…
• Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization
  28 Sep 2020 3 repositories listed Syntology ran 2 of 2 samples · 0 unverified
  In this paper, we introduce Apollo, a quasi-Newton method for nonconvex stochastic optimization, which dynamically incorporates the curvature of the loss function by approximating the Hessian via a diagonal matrix.

Syntology lines on 18 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.