{"url":"/method/sgd","slug":"sgd","name":"SGD","full_name":"Stochastic Gradient Descent","full_name_withheld":false,"description_markdown":"**Stochastic Gradient Descent** is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using all available data. That is for weights $w$ and a loss function $L$ we have:\r\n\r\n$$ w\\_{t+1} = w\\_{t} - \\eta\\hat{\\nabla}\\_{w}{L(w\\_{t})} $$\r\n\r\nWhere $\\eta$ is a learning rate. SGD reduces redundancy compared to batch gradient descent - which recomputes gradients for similar examples before each parameter update - so it is usually much faster.\r\n\r\n(Image Source: [here](http://rasbt.github.io/mlxtend/user_guide/general_concepts/gradient-optimization/))","description_state":"present","introduced_year":1951,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":"https://github.com/pytorch/pytorch/blob/4e0ac120e9a8b096069c2f892488d630a5c8f358/torch/optim/sgd.py#L97-L112","code_snippet_url_on_a_code_host":true,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":2021,"archive_num_papers":2021,"papers_newest_first":[{"paper":null,"title":"Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime","date":"2025-07-15","arxiv_id":"2507.11274","n_code_links":0,"syntology":null},{"paper":null,"title":"A Single Merging Suffices: Recovering Server-based Learning Performance in Decentralized Learning","date":"2025-07-09","arxiv_id":"2507.06542","n_code_links":0,"syntology":null},{"paper":null,"title":"Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learning","date":"2025-06-23","arxiv_id":"2506.18645","n_code_links":0,"syntology":null},{"paper":"/paper/a-minimalist-optimizer-design-for-llm","title":"A Minimalist Optimizer Design for LLM Pretraining","date":"2025-06-20","arxiv_id":"2506.16659","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":1}},{"paper":null,"title":"A Simplified Analysis of SGD for Linear Regression with Weight Averaging","date":"2025-06-18","arxiv_id":"2506.15535","n_code_links":0,"syntology":null},{"paper":"/paper/sharpness-aware-machine-unlearning","title":"Sharpness-Aware Machine Unlearning","date":"2025-06-16","arxiv_id":"2506.13715","n_code_links":0,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}},{"paper":null,"title":"Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling","date":"2025-06-14","arxiv_id":"2506.12543","n_code_links":0,"syntology":null},{"paper":null,"title":"Learning single-index models via harmonic decomposition","date":"2025-06-11","arxiv_id":"2506.09887","n_code_links":0,"syntology":null},{"paper":null,"title":"Improved Scaling Laws in Linear Regression via Data Reuse","date":"2025-06-10","arxiv_id":"2506.08415","n_code_links":0,"syntology":null},{"paper":null,"title":"Online Learning-guided Learning Rate Adaptation via Gradient Alignment","date":"2025-06-10","arxiv_id":"2506.08419","n_code_links":0,"syntology":null},{"paper":"/paper/an-adaptive-method-stabilizing-activations","title":"An Adaptive Method Stabilizing Activations for Enhanced Generalization","date":"2025-06-10","arxiv_id":"2506.08353","n_code_links":1,"syntology":null},{"paper":"/paper/2506-04430","title":"Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order","date":"2025-06-04","arxiv_id":"2506.04430","n_code_links":1,"syntology":null},{"paper":null,"title":"Classifying Dental Care Providers Through Machine Learning with Features Ranking","date":"2025-06-04","arxiv_id":"2506.04474","n_code_links":0,"syntology":null},{"paper":null,"title":"Incremental Gradient Descent with Small Epoch Counts is Surprisingly Slow on Ill-Conditioned Problems","date":"2025-06-04","arxiv_id":"2506.04126","n_code_links":0,"syntology":null},{"paper":null,"title":"Orthogonal Gradient Descent Improves Neural Calibration","date":"2025-06-04","arxiv_id":"2506.04487","n_code_links":0,"syntology":null},{"paper":null,"title":"Replay Can Provably Increase Forgetting","date":"2025-06-04","arxiv_id":"2506.04377","n_code_links":0,"syntology":null},{"paper":null,"title":"Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks","date":"2025-06-03","arxiv_id":"2506.02651","n_code_links":0,"syntology":null},{"paper":null,"title":"LightSAM: Parameter-Agnostic Sharpness-Aware Minimization","date":"2025-05-30","arxiv_id":"2505.24399","n_code_links":0,"syntology":null},{"paper":"/paper/sgd-as-free-energy-minimization-a","title":"SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training","date":"2025-05-29","arxiv_id":"2505.23489","n_code_links":1,"syntology":null},{"paper":null,"title":"The Rich and the Simple: On the Implicit Bias of Adam and SGD","date":"2025-05-29","arxiv_id":"2505.24022","n_code_links":0,"syntology":null},{"paper":null,"title":"DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models","date":"2025-05-28","arxiv_id":"2505.22549","n_code_links":0,"syntology":null},{"paper":null,"title":"PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning","date":"2025-05-28","arxiv_id":"2505.22085","n_code_links":0,"syntology":null},{"paper":null,"title":"AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent","date":"2025-05-27","arxiv_id":"2505.21651","n_code_links":0,"syntology":null},{"paper":null,"title":"Convergence of Clipped-SGD for Convex $(L_0,L_1)$-Smooth Optimization with Heavy-Tailed Noise","date":"2025-05-27","arxiv_id":"2505.20817","n_code_links":0,"syntology":null},{"paper":null,"title":"DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models","date":"2025-05-27","arxiv_id":"2505.21382","n_code_links":0,"syntology":null},{"paper":null,"title":"Gradient Flow Matching for Learning Update Dynamics in Neural Network Training","date":"2025-05-26","arxiv_id":"2505.20221","n_code_links":0,"syntology":null},{"paper":null,"title":"Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGD","date":"2025-05-24","arxiv_id":"2505.18535","n_code_links":0,"syntology":null},{"paper":null,"title":"Accelerating Learned Image Compression Through Modeling Neural Training Dynamics","date":"2025-05-23","arxiv_id":"2505.18107","n_code_links":0,"syntology":null},{"paper":"/paper/decomposition-of-water-demand-patterns-using","title":"Decomposition of Water Demand Patterns Using Skewed Gaussian Distributions for Behavioral Insights and Operational Planning","date":"2025-05-23","arxiv_id":"2505.18245","n_code_links":1,"syntology":null},{"paper":null,"title":"Emergence of Hebbian Dynamics in Regularized Non-Local Learners","date":"2025-05-23","arxiv_id":"2505.18069","n_code_links":0,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":149},{"task":"/task/image-classification","name":"Image Classification","papers":122},{"task":"/task/federated-learning","name":"Federated Learning","papers":110},{"task":"/task/image-classification","name":"image-classification","papers":102},{"task":"/task/machine-learning","name":"BIG-bench Machine Learning","papers":66},{"task":"/task/deep-learning","name":"Deep Learning","papers":66},{"task":"/task/regression-1","name":"regression","papers":59},{"task":"/task/quantization","name":"Quantization","papers":55},{"task":"/task/language-modelling","name":"Language Modelling","papers":52},{"task":null,"name":"GPU","papers":49},{"task":"/task/generalization-bounds","name":"Generalization Bounds","papers":45},{"task":"/task/language-modeling","name":"Language Modeling","papers":44},{"task":"/task/distributed-optimization","name":"Distributed Optimization","papers":39},{"task":"/task/classification","name":"General Classification","papers":39},{"task":"/task/computational-efficiency","name":"Computational Efficiency","papers":27},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":27},{"task":"/task/second-order-methods","name":"Second-order methods","papers":25},{"task":"/task/privacy-preserving","name":"Privacy Preserving","papers":24},{"task":"/task/learning-theory","name":"Learning Theory","papers":23},{"task":"/task/speech-recognition","name":"Speech Recognition","papers":23}],"tasks_shown":20,"n_tasks":362,"usage_by_year":[{"year":"2011","papers":3},{"year":"2012","papers":2},{"year":"2013","papers":10},{"year":"2014","papers":10},{"year":"2015","papers":25},{"year":"2016","papers":41},{"year":"2017","papers":75},{"year":"2018","papers":152},{"year":"2019","papers":218},{"year":"2020","papers":291},{"year":"2021","papers":337},{"year":"2022","papers":226},{"year":"2023","papers":270},{"year":"2024","papers":244},{"year":"2025","papers":116}],"row_source":"methods_table","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/sgd"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}