{"url":"/method/sgd-with-momentum","slug":"sgd-with-momentum","name":"SGD with Momentum","full_name":"SGD with Momentum","full_name_withheld":false,"description_markdown":"### Why SGD with Momentum?\r\nIn deep learning, we have used stochastic gradient descent as one of the optimizers because at the end we will find the minimum weight and bias at which the model loss is lowest. In the SGD we have some issues in which the SGD does not work perfectly because in deep learning we got a non-convex cost function graph and if use the simple SGD then it leads to low performance. There are 3 main reasons why it does not work:\r\n\r\n\"Non-convex\r\n\r\n1) We end up in local minima and not able to reach global minima\r\nAt the start, we randomly start at some point and we are going to end up at the local minimum and not able to reach the global minimum.\r\n\r\n2) Saddle Point will be the stop for reaching global minima\r\nA saddle point is a point where in one direction the surface goes in the upward direction and in another direction it goes downwards. So that the slope is changing very gradually so the speed of changing is going to slow and as result, the training also going to slow.\r\n\r\n3) High curvature can be a reason\r\nThe larger radius leads to low curvature and vice-versa. It will be difficult to traverse in the large curvature which was generally high in non-convex optimization.\r\nBy using the SGD with Momentum optimizer we can overcome the problems like high curvature, consistent gradient, and noisy gradient.\r\n\r\n### What is SGD with Momentum?\r\nSGD with Momentum is an optimization technique designed to improve the performance of neural networks. To understand the intuition behind it, consider the analogy of a ball rolling down a slope. As it rolls, the ball gains speed over time due to momentum.\r\n\r\nSuppose we have a starting point A and want to reach B. If we are uncertain about the correct direction, we can ask four other points that have already reached B. If all four indicate the same direction, we can move more confidently and at a faster pace. This is the key idea behind SGD with Momentum.\r\n\r\n\"Non-convex\r\n### How does SGD with Momentum work?\r\nTo understand its working, we must first explore the concept of the Exponentially Weighted Moving Average (EWMA), a technique used to identify trends in time-series data. The formula for EWMA is:\r\n\r\n\"Non-convex\r\n\r\n In the formula, β represents the weightage that is going to assign to the past values of the gradient. The values of β is from 0 < β < 1. If the value of the beta is 0.5 then it means that the 1/1–0.5 = 2 so it represents that the calculated average was from the previous 2 readings. \r\n\r\nThe value of Vt depends on β. The higher the value of β the more we try to get an average of more past data and vice-versa. For example, let's take the value of β 0.98 and 0.5 for two different scenarios so if we do 1/1-β then we get 50 and 10 respectively so it was clear that to calculate the average we take past 50 and 10 outcomes respectively for both cases.\r\nNow in SGD with Momentum, we use the same concept of EWMA. Here we introduce the term velocity v which is used to denote the change in the gradient to get to the global minima. The change in the weights is denoted by the formula:\r\n\r\n\"Non-convex\r\n\r\nthe β part of the V formula denotes and is useful to compute the confidence or we can say the past velocity for calculating Vt we have to calculate Vt-1 and for calculating Vt-1 we have to calculate Vt-2 and likewise. So we are using the history of velocity to calculate the momentum and this is the part that provides acceleration to the formula.\r\n\r\n\"Non-convex\r\n\r\nHere we have to consider two cases:\r\n1. β=0 then, as per the formula weight updating is going to just work as a Stochastic gradient descent. Here we called β a decaying factor because it is defining the speed of past velocity.\r\n\r\n2. β=1 then, there will be no decay. It involves the dynamic equilibrium which is not desired so we generally use the value of β like 0.9,0.99or 0.5 only.\r\n\r\n### Advantages of SGD with Momentum\r\n1. **Faster Convergence**: Momentum accelerates training, making it faster than standard SGD.\r\n2. **Escaping Local Minima**: Momentum helps the optimizer overcome small local minima and reach the global minimum.\r\n\r\n\"Non-convex\r\n\r\nIn the following visualization, purple represents SGD with Momentum, while light blue represents standard SGD. As shown, SGD with Momentum successfully reaches the global minimum, whereas standard SGD gets stuck in local minima.\r\n\r\nHowever, momentum itself can sometimes be problematic. When the optimizer reaches the global minimum, excessive momentum may cause oscillations before stabilizing, leading to longer training times. Despite this, SGD with Momentum remains significantly faster than standard SGD and is widely used in optimization.","description_state":"present","introduced_year":1999,"introduced_by":{"title":null,"paper":null,"first_author":null,"n_authors":0,"url_abs":null,"archive_paper_url":null},"source":{"url":null,"title":null,"url_on_a_paper_host":false},"code_snippet_url":null,"code_snippet_url_on_a_code_host":false,"categories":[{"area":"General","area_id":"general","collection":"Stochastic Optimization","url":"/methods/category/stochastic-optimization","pwc_aliases":[]}],"n_papers_tagged":144,"archive_num_papers":null,"papers_newest_first":[{"paper":"/paper/adams-momentum-itself-can-be-a-normalizer-for","title":"AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training","date":"2025-05-22","arxiv_id":"2505.16363","n_code_links":1,"syntology":{"ran":0,"of":1,"unverified":1,"pointer_only":0}},{"paper":null,"title":"Understanding Gradient Orthogonalization for Deep Learning via Non-Euclidean Trust-Region Optimization","date":"2025-03-16","arxiv_id":"2503.12645","n_code_links":0,"syntology":null},{"paper":null,"title":"Algorithmic Stability of Stochastic Gradient Descent with Momentum under Heavy-Tailed Noise","date":"2025-02-02","arxiv_id":"2502.00885","n_code_links":0,"syntology":null},{"paper":null,"title":"SOFIM: Stochastic Optimization Using Regularized Fisher Information Matrix","date":"2024-03-05","arxiv_id":"2403.02833","n_code_links":0,"syntology":null},{"paper":null,"title":"Should I try multiple optimizers when fine-tuning pre-trained Transformers for NLP tasks? Should I tune their hyperparameters?","date":"2024-02-10","arxiv_id":"2402.06948","n_code_links":0,"syntology":null},{"paper":null,"title":"Momentum Does Not Reduce Stochastic Noise in Stochastic Gradient Descent","date":"2024-02-04","arxiv_id":"2402.02325","n_code_links":0,"syntology":null},{"paper":"/paper/adaplus-integrating-nesterov-momentum-and","title":"AdaPlus: Integrating Nesterov Momentum and Precise Stepsize Adjustment on AdamW Basis","date":"2023-09-05","arxiv_id":"2309.01966","n_code_links":1,"syntology":null},{"paper":null,"title":"When and Why Momentum Accelerates SGD:An Empirical Study","date":"2023-06-15","arxiv_id":"2306.09000","n_code_links":0,"syntology":null},{"paper":"/paper/fast-diffusion-model","title":"Fast Diffusion Model","date":"2023-06-12","arxiv_id":"2306.06991","n_code_links":1,"syntology":{"ran":11,"of":17,"unverified":6,"pointer_only":0}},{"paper":null,"title":"Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning","date":"2023-05-23","arxiv_id":"2305.13664","n_code_links":0,"syntology":null},{"paper":"/paper/momo-momentum-models-for-adaptive-learning","title":"MoMo: Momentum Models for Adaptive Learning Rates","date":"2023-05-12","arxiv_id":"2305.07583","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}},{"paper":"/paper/a-view-of-mini-batch-sgd-via-generating","title":"A view of mini-batch SGD via generating functions: conditions of convergence, phase transitions, benefit from negative momenta","date":"2022-06-22","arxiv_id":"2206.11124","n_code_links":1,"syntology":{"ran":2,"of":2,"unverified":0,"pointer_only":0}},{"paper":"/paper/computing-the-variance-of-shuffling","title":"Computing the Variance of Shuffling Stochastic Gradient Algorithms via Power Spectral Density Analysis","date":"2022-06-01","arxiv_id":"2206.00632","n_code_links":1,"syntology":null},{"paper":"/paper/an-adaptive-gradient-method-with-energy-and","title":"An Adaptive Gradient Method with Energy and Momentum","date":"2022-03-23","arxiv_id":"2203.12191","n_code_links":1,"syntology":null},{"paper":null,"title":"Does Momentum Change the Implicit Regularization on Separable Data?","date":"2021-10-08","arxiv_id":"2110.03891","n_code_links":0,"syntology":null},{"paper":"/paper/using-a-one-dimensional-parabolic-model-of","title":"Using a one dimensional parabolic model of the full-batch loss to estimate learning rates during training","date":"2021-08-31","arxiv_id":"2108.13880","n_code_links":1,"syntology":null},{"paper":null,"title":"On the Hyperparameters in Stochastic Gradient Descent with Momentum","date":"2021-08-09","arxiv_id":"2108.03947","n_code_links":0,"syntology":null},{"paper":null,"title":"A New Adaptive Gradient Method with Gradient Decomposition","date":"2021-07-18","arxiv_id":"2107.08377","n_code_links":0,"syntology":null},{"paper":"/paper/lrtuner-a-learning-rate-tuner-for-deep-neural","title":"LRTuner: A Learning Rate Tuner for Deep Neural Networks","date":"2021-05-30","arxiv_id":"2105.14526","n_code_links":2,"syntology":null},{"paper":"/paper/fast-federated-learning-by-balancing","title":"Fast Federated Learning by Balancing Communication Trade-Offs","date":"2021-05-23","arxiv_id":"2105.11028","n_code_links":2,"syntology":null},{"paper":"/paper/positive-negative-momentum-manipulating","title":"Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization","date":"2021-03-31","arxiv_id":"2103.17182","n_code_links":1,"syntology":{"ran":1,"of":1,"unverified":0,"pointer_only":0}},{"paper":"/paper/bottleneck-transformers-for-visual","title":"Bottleneck Transformers for Visual Recognition","date":"2021-01-27","arxiv_id":"2101.11605","n_code_links":13,"syntology":{"ran":26,"of":49,"unverified":23,"pointer_only":8}},{"paper":"/paper/ikshana-a-theory-of-human-scene-understanding","title":"The Ikshana Hypothesis of Human Scene Understanding","date":"2021-01-21","arxiv_id":"2101.10837","n_code_links":2,"syntology":null},{"paper":"/paper/repvgg-making-vgg-style-convnets-great-again","title":"RepVGG: Making VGG-style ConvNets Great Again","date":"2021-01-11","arxiv_id":"2101.03697","n_code_links":25,"syntology":{"ran":13,"of":16,"unverified":3,"pointer_only":6}},{"paper":"/paper/magneto-an-efficient-deep-learning-method-for-1","title":"MAGNeto: An Efficient Deep Learning Method for the Extractive Tags Summarization Problem","date":"2020-11-09","arxiv_id":"2011.04349","n_code_links":1,"syntology":null},{"paper":null,"title":"Dual Averaging is Surprisingly Effective for Deep Learning Optimization","date":"2020-10-20","arxiv_id":"2010.10502","n_code_links":0,"syntology":null},{"paper":"/paper/aegd-adaptive-gradient-decent-with-energy","title":"AEGD: Adaptive Gradient Descent with Energy","date":"2020-10-10","arxiv_id":"2010.05109","n_code_links":1,"syntology":null},{"paper":"/paper/regularizing-neural-networks-via-adversarial","title":"Regularizing Neural Networks via Adversarial Model Perturbation","date":"2020-10-10","arxiv_id":"2010.04925","n_code_links":1,"syntology":{"ran":0,"of":12,"unverified":12,"pointer_only":0}},{"paper":"/paper/understanding-the-role-of-momentum-in-non","title":"Momentum via Primal Averaging: Theoretical Insights and Learning Rate Schedules for Non-Convex Optimization","date":"2020-10-01","arxiv_id":"2010.00406","n_code_links":1,"syntology":null},{"paper":"/paper/optimization-of-graph-neural-networks-with","title":"Optimization of Graph Neural Networks with Natural Gradient Descent","date":"2020-08-21","arxiv_id":"2008.09624","n_code_links":1,"syntology":null}],"papers_shown":30,"tasks":[{"task":"/task/image-classification","name":"Image Classification","papers":63},{"task":"/task/object-detection","name":"Object Detection","papers":51},{"task":"/task/object-detection-1","name":"object-detection","papers":28},{"task":"/task/image-classification","name":"image-classification","papers":26},{"task":"/task/classification","name":"General Classification","papers":23},{"task":"/task/object","name":"Object","papers":23},{"task":"/task/semantic-segmentation","name":"Semantic Segmentation","papers":23},{"task":"/task/instance-segmentation","name":"Instance Segmentation","papers":14},{"task":"/task/real-time-object-detection","name":"Real-Time Object Detection","papers":10},{"task":"/task/architecture-search","name":"Neural Architecture Search","papers":9},{"task":"/task/segmentation","name":"Segmentation","papers":9},{"task":"/task/classification-1","name":"Classification","papers":7},{"task":"/task/representation-learning","name":"Representation Learning","papers":7},{"task":"/task/image-segmentation","name":"Image Segmentation","papers":6},{"task":"/task/stochastic-optimization","name":"Stochastic Optimization","papers":6},{"task":"/task/data-augmentation","name":"Data Augmentation","papers":5},{"task":"/task/deep-learning","name":"Deep Learning","papers":5},{"task":"/task/domain-generalization","name":"Domain Generalization","papers":5},{"task":null,"name":"GPU","papers":5},{"task":"/task/object-recognition","name":"Object Recognition","papers":5}],"tasks_shown":20,"n_tasks":137,"usage_by_year":[{"year":"2012","papers":1},{"year":"2013","papers":2},{"year":"2014","papers":3},{"year":"2015","papers":10},{"year":"2016","papers":10},{"year":"2017","papers":12},{"year":"2018","papers":21},{"year":"2019","papers":42},{"year":"2020","papers":19},{"year":"2021","papers":10},{"year":"2022","papers":3},{"year":"2023","papers":5},{"year":"2024","papers":3},{"year":"2025","papers":3}],"row_source":"embedded","archive":{"source":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","archive_url":"https://paperswithcode.com/method/sgd-with-momentum"},"syntology_read_at":"2026-09-24T18:15:14+00:00"}