What Is the Steepest Descent Method?

The steepest descent method is an iterative way to find a minimum of a differentiable function. Starting from a point, it moves in the direction that gives the greatest immediate decrease in the function, then repeats the process.

The method is easy to implement, but its locally steepest direction is not always the fastest route to the minimum. The choice of step length matters, and some functions make the method converge very slowly.

The basic idea

For a differentiable function $f:\mathbb{R}^n\to\mathbb{R}$, the gradient $\nabla f(x)$ points in the direction of greatest local increase under the usual Euclidean distance. Therefore, the negative gradient, $-\nabla f(x)$, gives the direction of greatest local decrease.

The first-order approximation

$$f(x+td)\approx f(x)+t\nabla f(x)^T d$$

shows why. If $d=-\nabla f(x)$, then the linear change is $-t\|\nabla f(x)\|^2$, which is negative for a positive step length $t$ whenever the gradient is nonzero.

At iteration $k$, the update is

$$x_{k+1}=x_k-\alpha_k\nabla f(x_k),$$

where $\alpha_k>0$ is the step length, also called the learning rate. The direction is prescribed by the gradient; choosing the step length determines how far to travel along it.

How to apply the method

1. Choose an initial point $x_0$.
2. Calculate the gradient $\nabla f(x_k)$ at the current point.
3. Select a step length $\alpha_k$. It may be fixed, determined by a line search, or computed by minimizing $f(x_k-\alpha\nabla f(x_k))$ over $\alpha>0$.
4. Update the point using the steepest descent formula.
5. Repeat until a stopping condition is met, such as a sufficiently small gradient norm or a small change in the function value.

An exact line search chooses the step length that minimizes the function along the current direction. In practice, an exact minimum may be expensive to compute, so optimization algorithms often use an inexact line search that finds a step satisfying suitable decrease conditions.

Worked example

Consider

$$f(x,y)=(x-2)^2+2(y+1)^2$$

and start at $(x_0,y_0)=(0,0)$. Its gradient is

$$\nabla f(x,y)=\bigl(2(x-2),\,4(y+1)\bigr),$$

so at the starting point $\nabla f(0,0)=(-4,4)$. The steepest descent direction is therefore $(4,-4)$, and points along the line search have the form $(4\alpha,-4\alpha)$.

Substitute this into the objective:

$$f(4\alpha,-4\alpha)=(4\alpha-2)^2+2(-4\alpha+1)^2.$$

Differentiating with respect to $\alpha$ gives $96\alpha-32$. Setting this derivative to zero gives $\alpha=1/3$. The next point is thus

$$(x_1,y_1)=(0,0)-\frac13(-4,4)=\left(\frac43,-\frac43\right).$$

The function value falls from $f(0,0)=6$ to $f(4/3,-4/3)=2/3$. The minimum is at $(2,-1)$, so this step has reduced the objective but has not reached the minimum. The method computes a new gradient there and continues.

Connection to solving linear systems

Steepest descent is also used to solve $Ax=b$ when $A$ is symmetric positive definite. In that case, solving the system is equivalent to minimizing

$$\phi(x)=\frac12x^TAx-b^Tx.$$

Its gradient is $Ax-b$. If $r_k=b-Ax_k$ is the residual, then the negative gradient is $r_k$, so the update moves along the residual direction. With an exact line search, the step length is

$$\alpha_k=\frac{r_k^Tr_k}{r_k^TAr_k},$$

and $x_{k+1}=x_k+\alpha_k r_k$. This version is related to, but generally less efficient than, the conjugate gradient method for large symmetric positive-definite systems.

Common mistakes and limitations

• Using the gradient instead of its negative: For minimization, the update subtracts the gradient. Adding it moves uphill for a sufficiently small positive step.
• Assuming any step length will work well: A step that is too large can increase the function or cause oscillation; one that is too small makes progress painfully slow.
• Expecting the method to reach the global minimum: A zero gradient is a stationary point, which may be a local minimum, a maximum, or a saddle point. Convexity provides stronger guarantees about global minimization.
• Treating steepest as fastest overall: The direction is steepest only for the immediate local decrease measured with the Euclidean norm. On elongated or poorly scaled contours, successive steps can zigzag and make slow progress.

For a convex function, gradient-based methods can converge toward a global minimum under appropriate conditions on the step lengths. For a nonconvex function, the outcome can depend on the starting point. In either case, check the gradient norm and objective change rather than relying only on a fixed number of iterations.