Methods › General › Stochastic Optimization › SGD
Stochastic Gradient Descent
SGD
archive 2025-07-28 Description, source and code snippet are the archive's method entry.
Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using all available data. That is for weights w and a loss function L we have:
wₜ₊₁ = wₜ - η∇̂_wL(wₜ)
Where η is a learning rate. SGD reduces redundancy compared to batch gradient descent - which recomputes gradients for similar examples before each parameter update - so it is usually much faster.
(Image Source: here)
In Syntology name not resolved (unknown_name). Other entries in Syntology's graph carry this name; the lookup's note below says what that means for this name.
From the lookup's note on this name: Nothing resolved to this name, but 5 entries in Syntology's graph carry it; the most cited is the paper “SGD: General Analysis and Improved Rates” (arXiv:1901.09401, 488 citations). The lookup reads this as a miss of its own name matching, not a gap in coverage: the graph holds the name under a spelling the lookup did not reach. For arXiv:1901.09401: ask the lookup about it.
Near matches the lookup lists, each a different name: ∞-SGD; SGD-KV; INO-SGD; FBi-SGD; badr-sgd; FO-DP-SGD; DP-SGD-RC; SMA-DP-SGD.
Syntology's have check for this name (free, it answers coverage only), read 2026-10-05; ask it again live. A level is the served verification level of that one answer; nothing here is summed across methods.
Papers archive 2025-07-28
archive papers tagged: 2,021 · with a code link: 591 · where Syntology ran a sample: 192 (160 with a run with no instrument failure, 32 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (192 of 2,021 tagged: 160 with a run with no instrument failure, 32 where every run was a failure of Syntology's instrument)
Page 1 of 21: papers 1 to 100 of 2,021, newest first by the archive's date (ties by slug), in archive order.
Repository counts are the archive's code-links table. A Syntology line states what Syntology ran from that paper's harvested code, as “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. It is per sample and not a correctness claim. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from.
-
Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime 15 Jul 2025 · 0 repositories · arXiv:2507.11274
-
A Single Merging Suffices: Recovering Server-based Learning Performance in Decentralized Learning 9 Jul 2025 · 0 repositories · arXiv:2507.06542
-
Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learning 23 Jun 2025 · 0 repositories · arXiv:2506.18645
-
A Minimalist Optimizer Design for LLM Pretraining 20 Jun 2025 · 1 repository · arXiv:2506.16659Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
A Simplified Analysis of SGD for Linear Regression with Weight Averaging 18 Jun 2025 · 0 repositories · arXiv:2506.15535
-
Sharpness-Aware Machine Unlearning 16 Jun 2025 · 0 repositories · arXiv:2506.13715Syntology 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample)
-
Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling 14 Jun 2025 · 0 repositories · arXiv:2506.12543
-
Learning single-index models via harmonic decomposition 11 Jun 2025 · 0 repositories · arXiv:2506.09887
-
Improved Scaling Laws in Linear Regression via Data Reuse 10 Jun 2025 · 0 repositories · arXiv:2506.08415
-
Online Learning-guided Learning Rate Adaptation via Gradient Alignment 10 Jun 2025 · 0 repositories · arXiv:2506.08419
-
An Adaptive Method Stabilizing Activations for Enhanced Generalization 10 Jun 2025 · 1 repository · arXiv:2506.08353
-
Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order 4 Jun 2025 · 1 repository · arXiv:2506.04430
-
Classifying Dental Care Providers Through Machine Learning with Features Ranking 4 Jun 2025 · 0 repositories · arXiv:2506.04474
-
Incremental Gradient Descent with Small Epoch Counts is Surprisingly Slow on Ill-Conditioned Problems 4 Jun 2025 · 0 repositories · arXiv:2506.04126
-
Orthogonal Gradient Descent Improves Neural Calibration 4 Jun 2025 · 0 repositories · arXiv:2506.04487
-
Replay Can Provably Increase Forgetting 4 Jun 2025 · 0 repositories · arXiv:2506.04377
-
Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks 3 Jun 2025 · 0 repositories · arXiv:2506.02651
-
LightSAM: Parameter-Agnostic Sharpness-Aware Minimization 30 May 2025 · 0 repositories · arXiv:2505.24399
-
SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training 29 May 2025 · 1 repository · arXiv:2505.23489
-
The Rich and the Simple: On the Implicit Bias of Adam and SGD 29 May 2025 · 0 repositories · arXiv:2505.24022
-
DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models 28 May 2025 · 0 repositories · arXiv:2505.22549
-
PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning 28 May 2025 · 0 repositories · arXiv:2505.22085
-
AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent 27 May 2025 · 0 repositories · arXiv:2505.21651
-
Convergence of Clipped-SGD for Convex (L₀,L₁)-Smooth Optimization with Heavy-Tailed Noise 27 May 2025 · 0 repositories · arXiv:2505.20817
-
DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models 27 May 2025 · 0 repositories · arXiv:2505.21382
-
Gradient Flow Matching for Learning Update Dynamics in Neural Network Training 26 May 2025 · 0 repositories · arXiv:2505.20221
-
Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGD 24 May 2025 · 0 repositories · arXiv:2505.18535
-
Accelerating Learned Image Compression Through Modeling Neural Training Dynamics 23 May 2025 · 0 repositories · arXiv:2505.18107
-
Decomposition of Water Demand Patterns Using Skewed Gaussian Distributions for Behavioral Insights and Operational Planning 23 May 2025 · 1 repository · arXiv:2505.18245
-
Emergence of Hebbian Dynamics in Regularized Non-Local Learners 23 May 2025 · 0 repositories · arXiv:2505.18069
-
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training 22 May 2025 · 1 repository · arXiv:2505.16363Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)
-
Dimension-adapted Momentum Outscales SGD 22 May 2025 · 0 repositories · arXiv:2505.16098
-
Smoothed SGD for quantiles: Bahadur representation and Gaussian approximation 19 May 2025 · 0 repositories · arXiv:2505.13299
-
Accelerating Neural Network Training Along Sharp and Flat Directions 17 May 2025 · 0 repositories · arXiv:2505.11972
-
On the O((√(d))/(K^(1/4))) Convergence Rate of AdamW Measured by ℓ₁ Norm 17 May 2025 · 0 repositories · arXiv:2505.11840
-
Revisiting Stochastic Approximation and Stochastic Gradient Descent 16 May 2025 · 0 repositories · arXiv:2505.11343
-
Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization 16 May 2025 · 0 repositories · arXiv:2505.11434
-
DeepSeqCoco: A Robust Mobile Friendly Deep Learning Model for Detection of Diseases in Cocos nucifera 15 May 2025 · 0 repositories · arXiv:2505.10030
-
Birch SGD: A Tree Graph Framework for Local and Asynchronous SGD Methods 14 May 2025 · 0 repositories · arXiv:2505.09218
-
Differentiable Channel Selection in Self-Attention For Person Re-Identification 13 May 2025 · 1 repository · arXiv:2505.08961
-
Rapid Overfitting of Multi-Pass Stochastic Gradient Descent in Stochastic Convex Optimization 13 May 2025 · 0 repositories · arXiv:2505.08306
-
Sharp Gaussian approximations for Decentralized Federated Learning 12 May 2025 · 0 repositories · arXiv:2505.08125
-
Streaming Krylov-Accelerated Stochastic Gradient Descent 11 May 2025 · 0 repositories · arXiv:2505.07046
-
Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness 7 May 2025 · 0 repositories · arXiv:2505.04599
-
More Optimal Fractional-Order Stochastic Gradient Descent for Non-Convex Optimization Problems 5 May 2025 · 0 repositories · arXiv:2505.02985
-
Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients 3 May 2025 · 0 repositories · arXiv:2505.01744
-
Emergence and scaling laws in SGD learning of shallow neural networks 28 Apr 2025 · 0 repositories · arXiv:2504.19983
-
Theoretical Framework for Tempered Fractional Gradient Descent: Application to Breast Cancer Classification 26 Apr 2025 · 0 repositories · arXiv:2504.18849
-
Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis 26 Apr 2025 · 1 repository · arXiv:2504.20096
-
Learning Operators by Regularized Stochastic Gradient Descent with Operator-valued Kernels 25 Apr 2025 · 0 repositories · arXiv:2504.18184
-
Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness 21 Apr 2025 · 0 repositories · arXiv:2504.14882
-
Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methods 20 Apr 2025 · 0 repositories · arXiv:2504.14701
-
First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms 18 Apr 2025 · 0 repositories · arXiv:2504.13992
-
Stochastic Gradient Descent in Non-Convex Problems: Asymptotic Convergence with Relaxed Step-Size via Stopping Time Methods 17 Apr 2025 · 0 repositories · arXiv:2504.12601
-
Corner Gradient Descent 16 Apr 2025 · 1 repository · arXiv:2504.12519
-
A Piecewise Lyapunov Analysis of Sub-quadratic SGD: Applications to Robust and Quantile Regression 11 Apr 2025 · 0 repositories · arXiv:2504.08178
-
Better Rates for Random Task Orderings in Continual Linear Models 6 Apr 2025 · 0 repositories · arXiv:2504.04579
-
Architect Your Landscape Approach (AYLA) for Optimizations in Deep Learning 2 Apr 2025 · 0 repositories · arXiv:2504.01875
-
Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions 31 Mar 2025 · 0 repositories · arXiv:2503.23896
-
Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descent 31 Mar 2025 · 0 repositories · arXiv:2503.23642
-
Nested Stochastic Gradient Descent for (Generalized) Sinkhorn Distance-Regularized Distributionally Robust Optimization 29 Mar 2025 · 0 repositories · arXiv:2503.22923
-
Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent 28 Mar 2025 · 0 repositories · arXiv:2503.22478
-
Streaming Federated Learning with Markovian Data 24 Mar 2025 · 0 repositories · arXiv:2503.18807
-
Finding Stable Subnetworks at Initialization with Dataset Distillation 23 Mar 2025 · 0 repositories · arXiv:2503.17905
-
The global convergence time of stochastic gradient descent in non-convex landscapes: Sharp estimates via large deviations 20 Mar 2025 · 0 repositories · arXiv:2503.16398
-
Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learning 18 Mar 2025 · 0 repositories · arXiv:2503.14205
-
Effective Dimension Aware Fractional-Order Stochastic Gradient Descent for Convex Optimization Problems 17 Mar 2025 · 0 repositories · arXiv:2503.13764
-
Convergence Analysis of alpha-SVRG under Strong Convexity 16 Mar 2025 · 0 repositories · arXiv:2503.12454
-
Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization 16 Mar 2025 · 0 repositories · arXiv:2503.12645
-
ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory 16 Mar 2025 · 1 repository · arXiv:2503.12668
-
Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning 14 Mar 2025 · 0 repositories · arXiv:2503.11147
-
Global Convergence and Rich Feature Learning in L-Layer Infinite-Width Neural Networks under μP Parametrization 12 Mar 2025 · 0 repositories · arXiv:2503.09565
-
Water Quality Data Imputation via A Fast Latent Factorization of Tensors with PID-based Optimizer 10 Mar 2025 · 0 repositories · arXiv:2503.06997
-
Analyzing the Role of Permutation Invariance in Linear Mode Connectivity 8 Mar 2025 · 0 repositories · arXiv:2503.06001
-
Efficiently Escaping Saddle Points under Generalized Smoothness via Self-Bounding Regularity 6 Mar 2025 · 0 repositories · arXiv:2503.04712
-
Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approach 6 Mar 2025 · 0 repositories · arXiv:2503.04918
-
Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks 4 Mar 2025 · 1 repository · arXiv:2503.02147
-
Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networks 3 Mar 2025 · 0 repositories · arXiv:2503.01660
-
On the Byzantine Fault Tolerance of signSGD with Majority Vote 26 Feb 2025 · 0 repositories · arXiv:2502.19170
-
An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Losses 25 Feb 2025 · 0 repositories · arXiv:2502.17772
-
SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation 25 Feb 2025 · 1 repository · arXiv:2502.18153Syntology official (archive's flag): 3 ran · 3 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs 24 Feb 2025 · 0 repositories · arXiv:2502.17009
-
A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Models 20 Feb 2025 · 1 repository · arXiv:2502.15828Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 4 harvested samples) · 4 pointer-only (licence)
-
Variance Reduction Methods Do Not Need to Compute Full Gradients: Improved Efficiency through Shuffling 20 Feb 2025 · 0 repositories · arXiv:2502.14648
-
Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression 19 Feb 2025 · 0 repositories · arXiv:2502.14123
-
Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback 18 Feb 2025 · 0 repositories · arXiv:2502.13298
-
LiveVal: Time-aware Data Valuation via Adaptive Reference Points 14 Feb 2025 · 0 repositories · arXiv:2502.10489
-
SeWA: Selective Weight Average via Probabilistic Masking 14 Feb 2025 · 0 repositories · arXiv:2502.10119
-
Linear-Time User-Level DP-SCO via Robust Statistics 13 Feb 2025 · 0 repositories · arXiv:2502.08889
-
Optimal Algorithms in Linear Regression under Covariate Shift: On the Importance of Precondition 13 Feb 2025 · 0 repositories · arXiv:2502.09047
-
Scaling Law for Stochastic Gradient Descent in Quadratically Parameterized Linear Regression 13 Feb 2025 · 0 repositories · arXiv:2502.09106
-
Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension 11 Feb 2025 · 0 repositories · arXiv:2502.07752
-
Analog In-memory Training on General Non-ideal Resistive Elements: The Impact of Response Functions 10 Feb 2025 · 0 repositories · arXiv:2502.06309
-
Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descent 10 Feb 2025 · 0 repositories · arXiv:2502.06719
-
Satellite Observations Guided Diffusion Model for Accurate Meteorological States at Arbitrary Resolution 9 Feb 2025 · 0 repositories · arXiv:2502.07814
-
The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networks 8 Feb 2025 · 0 repositories · arXiv:2502.05668
-
Data-Parallel Neural Network Training via Nonlinearly Preconditioned Trust-Region Method 7 Feb 2025 · 0 repositories · arXiv:2502.05133
-
Bayesian Parameter Shift Rule in Variational Quantum Eigensolvers 4 Feb 2025 · 0 repositories · arXiv:2502.02625
-
On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning 3 Feb 2025 · 0 repositories · arXiv:2502.01763
-
Training in reverse: How iteration order influences convergence and stability in deep learning 3 Feb 2025 · 0 repositories · arXiv:2502.01557
Tasks archive 2025-07-28
20 shown of 362 tasks the archive attaches to papers tagged with this method, by distinct papers. A task without a page in the catalog is plain text.
Usage over time archive 2025-07-28
Components: the archive holds no method-to-method composition, so PwC's Components table cannot be rebuilt; the Papers list carries no Results column for the same reason (the archive does not join its leaderboard rows to method tags).
Categories archive 2025-07-28
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections