SGD with Momentum

Why SGD with Momentum?

In deep learning, we have used stochastic gradient descent as one of the optimizers because at the end we will find the minimum weight and bias at which the model loss is lowest. In the SGD we have some issues in which the SGD does not work perfectly because in deep learning we got a non-convex cost function graph and if use the simple SGD then it leads to low performance. There are 3 main reasons why it does not work:

[Non-convex graph]

1) We end up in local minima and not able to reach global minima At the start, we randomly start at some point and we are going to end up at the local minimum and not able to reach the global minimum.

2) Saddle Point will be the stop for reaching global minima A saddle point is a point where in one direction the surface goes in the upward direction and in another direction it goes downwards. So that the slope is changing very gradually so the speed of changing is going to slow and as result, the training also going to slow.

3) High curvature can be a reason The larger radius leads to low curvature and vice-versa. It will be difficult to traverse in the large curvature which was generally high in non-convex optimization. By using the SGD with Momentum optimizer we can overcome the problems like high curvature, consistent gradient, and noisy gradient.

What is SGD with Momentum?

SGD with Momentum is an optimization technique designed to improve the performance of neural networks. To understand the intuition behind it, consider the analogy of a ball rolling down a slope. As it rolls, the ball gains speed over time due to momentum.

Suppose we have a starting point A and want to reach B. If we are uncertain about the correct direction, we can ask four other points that have already reached B. If all four indicate the same direction, we can move more confidently and at a faster pace. This is the key idea behind SGD with Momentum.

[Non-convex graph]

How does SGD with Momentum work?

To understand its working, we must first explore the concept of the Exponentially Weighted Moving Average (EWMA), a technique used to identify trends in time-series data. The formula for EWMA is:

[Non-convex graph]

In the formula, β represents the weightage that is going to assign to the past values of the gradient. The values of β is from 0 < β < 1. If the value of the beta is 0.5 then it means that the 1/1–0.5 = 2 so it represents that the calculated average was from the previous 2 readings.

The value of Vt depends on β. The higher the value of β the more we try to get an average of more past data and vice-versa. For example, let's take the value of β 0.98 and 0.5 for two different scenarios so if we do 1/1-β then we get 50 and 10 respectively so it was clear that to calculate the average we take past 50 and 10 outcomes respectively for both cases. Now in SGD with Momentum, we use the same concept of EWMA. Here we introduce the term velocity v which is used to denote the change in the gradient to get to the global minima. The change in the weights is denoted by the formula:

[Non-convex graph]

the β part of the V formula denotes and is useful to compute the confidence or we can say the past velocity for calculating Vt we have to calculate Vt-1 and for calculating Vt-1 we have to calculate Vt-2 and likewise. So we are using the history of velocity to calculate the momentum and this is the part that provides acceleration to the formula.

[Non-convex graph]

Here we have to consider two cases: 1. β=0 then, as per the formula weight updating is going to just work as a Stochastic gradient descent. Here we called β a decaying factor because it is defining the speed of past velocity.

1. β=1 then, there will be no decay. It involves the dynamic equilibrium which is not desired so we generally use the value of β like 0.9,0.99or 0.5 only.

Advantages of SGD with Momentum

1. Faster Convergence: Momentum accelerates training, making it faster than standard SGD.
2. Escaping Local Minima: Momentum helps the optimizer overcome small local minima and reach the global minimum.

[Non-convex graph]

In the following visualization, purple represents SGD with Momentum, while light blue represents standard SGD. As shown, SGD with Momentum successfully reaches the global minimum, whereas standard SGD gets stuck in local minima.

However, momentum itself can sometimes be problematic. When the optimizer reaches the global minimum, excessive momentum may cause oscillations before stabilizing, leading to longer training times. Despite this, SGD with Momentum remains significantly faster than standard SGD and is widely used in optimization.