How to Write Least Squares in Matrix Form

Least squares is a way to find the best approximate solution when a matrix equation has no exact solution, as commonly happens when fitting a model to more observations than unknown coefficients. Its matrix form is a minimization problem: choose the coefficient vector that makes the residual as small as possible.

This article sets up the problem, derives the normal equations, and works through a straight-line fit. The closed-form formula is useful for understanding the method, while QR factorization and singular value decomposition are generally preferable for numerical computation.

The matrix formulation

Let $A$ be an $m\times n$ matrix, $x\in\mathbb{R}^n$ the unknown coefficient vector, and $b\in\mathbb{R}^m$ the observed data vector. The system is

$$Ax=b.$$

When $m>n$, there are more equations than unknowns, and the equations may be inconsistent. The residual for a proposed $x$ is $r=b-Ax$. The least-squares solution, written $\hat{x}$, minimizes the squared Euclidean length of that residual:

$$\hat{x}=\operatorname*{arg\,min}_{x\in\mathbb{R}^n}\|Ax-b\|_2^2.$$

Squaring the norm does not change which $x$ minimizes it, but makes the objective convenient to differentiate. The entries of $A$ encode the model, the entries of $x$ are the coefficients to estimate, and the entries of $b$ are the measured values.

Deriving the normal equations

Expand the objective as a function of $x$:

$$f(x)=\|Ax-b\|_2^2=(Ax-b)^T(Ax-b).$$

Its gradient is $\nabla f(x)=2A^T(Ax-b)$. At a minimum, the gradient is zero, so

$$A^T(A\hat{x}-b)=0,$$

or equivalently,

$$A^TA\hat{x}=A^Tb.$$

These are the normal equations. If the columns of $A$ are linearly independent, then $A^TA$ is invertible, giving the familiar formula

$$\hat{x}=(A^TA)^{-1}A^Tb.$$

This formula is a theoretical expression, not usually the best way to calculate the answer: explicitly forming an inverse can magnify numerical error. For numerical work, solve the least-squares problem with QR factorization or SVD instead.

Geometrically, $A\hat{x}$ is the point in the column space of $A$ closest to $b$. The residual is perpendicular to that column space, which is why $A^Tr=0$. This orthogonality condition is another way to understand the normal equations.

Worked example: fitting a straight line

Fit a line $y=c+mx$ to the points $(0,1)$, $(1,2)$, and $(2,2)$ by minimizing the sum of squared vertical errors. The unknown vector is $x=[c\;m]^T$, and the data give

$$A=\begin{bmatrix}1&0\\1&1\\1&2\end{bmatrix},\qquad b=\begin{bmatrix}1\\2\\2\end{bmatrix}.$$

Each row of $A$ contains the coefficients of $c$ and $m$ for one observation. Calculate the two sides of the normal equations:

$$A^TA=\begin{bmatrix}3&3\\3&5\end{bmatrix},\qquad A^Tb=\begin{bmatrix}5\\6\end{bmatrix}.$$

Thus $3c+3m=5$ and $3c+5m=6$. Subtracting the first equation from the second gives $2m=1$, so $m=1/2$ and $c=7/6$. The fitted line is

$$\hat{y}=\frac{7}{6}+\frac{1}{2}x.$$

Its predictions at the three observed $x$-values are $7/6$, $5/3$, and $13/6$. Therefore the residual vector, using $r=b-A\hat{x}$, is

$$r=\begin{bmatrix}-1/6\\1/3\\-1/6\end{bmatrix},\qquad \|r\|_2^2=\frac{1}{6}.$$

As a check, the residuals sum to zero and their products with the corresponding $x$-values also sum to zero. In matrix terms, $A^Tr=0$, confirming that the residual is orthogonal to both columns of $A$.

When the formula needs care

The expression $(A^TA)^{-1}A^Tb$ requires $A$ to have full column rank: its columns must be linearly independent. If they are dependent, there can be multiple coefficient vectors with the same best fit, and $A^TA$ is singular. The Moore–Penrose pseudoinverse gives the minimum-norm least-squares solution, while SVD makes rank deficiencies easier to identify.

For a weighted least-squares fit, where observations have different weights, minimize $(Ax-b)^TW(Ax-b)$ for a suitable weight matrix $W$. If $W$ is symmetric positive definite, the corresponding normal equations are $A^TWAx=A^TWb$. Ordinary least squares is the special case $W=I$.

Common mistakes

• Trying to invert a rectangular matrix: A non-square $A$ has no ordinary matrix inverse. Least squares instead solves a minimization problem.
• Using the closed-form inverse without checking rank: The formula involving $(A^TA)^{-1}$ applies only when $A$ has full column rank.
• Assuming the fit has zero residual: A least-squares solution need not satisfy $A\hat{x}=b$. It minimizes the residual; it does not necessarily eliminate it.
• Confusing residual conventions: This article uses $r=b-A\hat{x}$. The alternative $A\hat{x}-b$ changes the sign, but not the residual norm or the least-squares solution.
• Using normal equations blindly in computation: When $A$ is poorly conditioned, solving with QR or SVD is generally more stable than forming $A^TA$.