9.6 Gradient Descent

From optimization to an algorithm

In the previous section, we reformulated supervised learning as an optimization problem. Our goal is no longer to find the “best regression line” directly. Instead, we seek the model parameters that minimize the loss function.

This naturally leads to the next question:

How can we systematically find the minimum of the loss function?

For the simple linear regression model introduced in this chapter, this problem has an exact analytical solution. By differentiating the Mean Squared Error with respect to the model parameters and setting the derivative equal to zero, the optimal regression coefficients can be calculated directly.

However, this strategy quickly becomes impractical for more complex models.

Modern machine learning algorithms often contain millions or even billions of adjustable parameters. For such models, analytical solutions are either computationally infeasible or do not exist at all.

Instead, we require an algorithm that gradually improves the model until the prediction error can no longer be reduced.

The most important optimization algorithm for this purpose is gradient descent.

Remarkably, although today’s machine learning models have become enormously sophisticated, the central optimization principle has changed very little. The same algorithm that optimizes a simple linear regression also forms the basis for training modern deep neural networks.

The gradient: measuring the local slope

The key idea behind gradient descent comes from differential calculus.

Suppose we stand at some point on the loss landscape. We may not know where the global minimum is located, but we can ask a simpler question:

If we move slightly from our current position, how does the loss change?

The answer is provided by the gradient.

For a function of a single variable, the gradient reduces to the ordinary derivative,

\frac{d\,\mathrm{MSE}}{dm},

which describes the slope of the loss function at the current parameter value.

The derivative provides two pieces of information simultaneously.

First, its sign indicates the direction in which the loss increases.

A positive derivative means that the loss increases when the parameter becomes larger.
A negative derivative means that the loss decreases when the parameter becomes larger.

Second, its magnitude indicates how rapidly the loss changes.

A large derivative corresponds to a steep slope, whereas a small derivative indicates a relatively flat region of the loss landscape.

Thus, the derivative tells us exactly how the prediction error behaves in the immediate neighborhood of the current model.

A geometric interpretation

The gradient can be understood intuitively using the analogy of a mountain landscape.

Imagine standing somewhere on a hillside.

You cannot see the entire landscape because dense fog limits your view. Nevertheless, you can immediately determine the direction of the steepest ascent by looking at the local slope beneath your feet.

The gradient plays exactly this role.

At every point of the loss landscape, it points in the direction of the steepest increase of the loss function.

Its length reflects how steep the slope is.

This interpretation immediately reveals why gradient descent works.

Our objective is not to climb the mountain.

We wish to descend into the valley where the prediction error is minimal.

Consequently, we always move opposite to the gradient.

This is the origin of the name gradient descent.

The gradient indicates the direction of steepest ascent, while gradient descent follows the direction of steepest descent.

Updating the model parameters

Gradient descent improves the model iteratively.

Starting from an initial guess for the model parameters, the algorithm repeatedly performs the same sequence of operations.

First, it computes the gradient of the loss function at the current parameter values.

Second, it updates the parameters by moving a small step opposite to the gradient.

Finally, it repeats this procedure until further improvements become negligible.

Mathematically, the parameter update is written as

m_{\text{new}} = m_{\text{old}} - \alpha \frac{d\,\mathrm{MSE}}{dm},

where

(m_{\text{old}}) is the current parameter,
(\frac{d,\mathrm{MSE}}{dm}) is the gradient,
and (\alpha) is the learning rate.

Although this equation is compact, it summarizes the entire learning process.

Every iteration modifies the model slightly.

If the update reduces the prediction error, the new parameter becomes the starting point for the next iteration.

Over time, these small improvements accumulate until the model converges toward the minimum of the loss function.

The role of the learning rate

The learning rate,

\alpha,

controls the size of each optimization step.

Its value strongly influences how efficiently the model learns.

If the learning rate is very small, the algorithm makes only tiny parameter updates.

This usually guarantees stable convergence but may require many thousands of iterations before the optimum is reached.

Conversely, a very large learning rate allows the algorithm to move rapidly through parameter space.

While this may accelerate learning initially, excessively large updates introduce a new problem.

Instead of approaching the minimum smoothly, the algorithm may repeatedly overshoot it.

Rather than converging, the parameter oscillates from one side of the minimum to the other.

If the learning rate becomes too large, the optimization may even diverge, causing the prediction error to increase rather than decrease.

Choosing an appropriate learning rate therefore represents a balance between speed and stability.

This remains one of the most important practical considerations in training machine learning models.

Convergence: when has learning finished?

Gradient descent gradually improves the model by reducing the prediction error at each iteration.

Eventually, the algorithm reaches a point where further updates produce only negligible improvements.

At this stage, the gradient approaches zero because the loss landscape has become locally flat.

The algorithm is then said to have converged.

Convergence does not necessarily imply that the model perfectly describes biological reality.

Rather, it indicates that, given the chosen model structure and the available data, no nearby parameter values yield substantially better predictions.

For our simple regression example, convergence identifies the regression line that minimizes the Mean Squared Error.

For modern neural networks, exactly the same principle applies.

The only difference is that millions of parameters are updated simultaneously instead of only one or two.

Thus, gradient descent scales naturally from simple regression models to the largest machine learning systems currently in use.

Learning through incremental improvement

Perhaps the most remarkable aspect of gradient descent is its simplicity.

At no point does the algorithm attempt to solve the entire optimization problem at once.

Instead, it repeatedly asks a single local question:

How should the current model be changed to reduce the prediction error most effectively?

Each individual update is usually small.

Nevertheless, thousands or millions of these incremental improvements collectively transform an initially poor model into an accurate predictor.

This illustrates a fundamental principle of machine learning.

Learning is not a single event.

It is a process of continuous refinement.

Each iteration incorporates a little more information from the experimental data, gradually improving the model’s ability to capture the underlying biological relationship.

Whether the model contains one parameter or one billion, the learning mechanism remains exactly the same.

Learning Goals

After completing this section, you should be able to

explain why gradient descent is needed for modern machine learning,
interpret the gradient as the local slope of the loss landscape,
understand why optimization proceeds opposite to the gradient,
explain the parameter update rule of gradient descent,
describe the influence of the learning rate,
explain what convergence means in the context of supervised learning.

Conceptual Summary

Gradient descent is the most widely used optimization algorithm in machine learning. Rather than solving the optimization problem analytically, it improves the model iteratively by repeatedly moving the parameters in the direction that most strongly decreases the loss function. The gradient provides this direction locally, while the learning rate determines the size of each update. Although introduced here for a simple regression model, the same optimization principle underlies the training of virtually all modern machine learning models, including deep neural networks.

Self-Check Questions

Why is gradient descent needed even though linear regression has an analytical solution?
What information does the gradient provide?
Why does gradient descent move opposite to the gradient?
What role does the learning rate play during optimization?
What happens if the learning rate is chosen too small? Too large?
What does it mean for gradient descent to converge?
Why can the same optimization algorithm be applied to both linear regression and deep neural networks?