What Is the Intuitive Relationship Between SVD and PCA?

PCA and the singular value decomposition (SVD) are closely connected, but they are not the same thing. PCA is a way to find directions that capture as much variation in a dataset as possible. SVD is a matrix factorization that, when applied to the appropriately prepared data matrix, gives those directions directly.

The key condition is that the data are usually centered first: subtract the mean of each feature from its observations. The explanation below uses observations in rows and features in columns; transposing the data changes which side of the SVD contains the principal directions.

The connection in one equation

Let $X$ be an $n \times p$ matrix of centered data, with $n$ observations and $p$ features. Its SVD is

$$X = U\Sigma V^T.$$

The columns of $V$ are directions in feature space. For centered data, these are the principal component directions. The singular values on the diagonal of $\Sigma$ tell you how much variation lies along each direction: their squares are proportional to the component variances.

The component scores—the coordinates of the observations in the new PCA directions—are

$$XV = U\Sigma.$$

So, in practical terms, the right singular vectors give the PCA directions, and the left singular vectors multiplied by the singular values give the scores.

Why this works

The sample covariance matrix of centered data is

$$C = \frac{1}{n-1}X^TX.$$

Substitute the SVD, using $U^TU=I$:

$$C = \frac{1}{n-1}V\Sigma^T\Sigma V^T.$$

Thus, the columns of $V$ are eigenvectors of the covariance matrix, and the corresponding eigenvalues are $\sigma_i^2/(n-1)$, where $\sigma_i$ is the corresponding singular value. Those eigenvectors are precisely the directions PCA chooses, ordered from greatest variance to least.

There is also a geometric way to see it. Pick a unit direction $v$ in feature space. Projecting every row of $X$ onto it gives the score vector $Xv$. Its squared length is

$$\lVert Xv\rVert^2 = v^T X^T Xv.$$

PCA chooses the unit direction that makes this quantity as large as possible. The eigenvector with the largest eigenvalue of $X^TX$ maximizes it; the SVD supplies that eigenvector as the first column of $V$. PCA then finds the next direction subject to being perpendicular to the first, and continues.

A worked example

Consider three observations on two features:

$$X = \begin{bmatrix} -1 & -1 \\ 0 & 2 \\ 1 & -1 \end{bmatrix}.$$

Both columns already have mean zero. Their cross-product matrix is

$$X^TX = \begin{bmatrix}2 & 0 \\ 0 & 6\end{bmatrix}.$$

The larger eigenvalue is $6$, along the second feature axis; the other is $2$, along the first. Accordingly, the singular values are $\sqrt{6}$ and $\sqrt{2}$, and the right singular vectors are the two feature axes, ordered second feature first.

Projecting the data onto those directions gives scores

$$XV = \begin{bmatrix} -1 & -1 \\ 2 & 0 \\ -1 & 1 \end{bmatrix},$$

where the first score column corresponds to the second original feature. Using the sample covariance convention, the component variances are $6/(3-1)=3$ and $2/(3-1)=1$. The first component therefore accounts for $3/(3+1)=75\%$ of the total variance, and the second accounts for $25\%$.

What changes with scaling or conventions?

Dividing covariance by $n-1$ gives sample variances; dividing by $n$ gives population variances. This changes the numerical variance values by a common factor, but not the principal directions or the fractions of variance explained.

If features have very different units or scales, you may standardize each feature before PCA. That means PCA is then performed on the centered, scaled data (equivalently, a correlation-based analysis), so the SVD directions can differ from those obtained using unscaled data. Whether to scale depends on what differences between features are meaningful for your problem.

Common mistakes

• Applying SVD to uncentered data and calling it ordinary PCA. Without centering, the leading direction may reflect the data’s offset from zero rather than variation around its mean.
• Confusing singular values with variances. The variance along a component is proportional to $\sigma_i^2$, not $\sigma_i$. For sample covariance, it is $\sigma_i^2/(n-1)$.
• Assuming SVD and PCA are identical in every use. SVD factors any matrix; PCA is a statistical analysis with choices such as centering and scaling. SVD gives the PCA result when applied to the suitably prepared data matrix.
• Treating a component’s sign as meaningful. If a singular vector and its scores are both multiplied by $-1$, the same component is represented. The direction and its reverse describe the same axis.
• Expecting a unique direction when values are tied. If singular values are repeated, the individual directions within that tied subspace may not be unique, even though the subspace is.

In computation, SVD is often used to obtain PCA because it works directly on the data matrix and avoids explicitly forming the covariance matrix. The underlying relationship remains the same: covariance eigenvectors are right singular vectors, and covariance eigenvalues are squared singular values scaled by the variance convention.