Ma · Mathematics

How to Compute a Covariance Matrix from an SVD in NumPy

Published By the 9to5Science editorial team

At a glance

  1. 1. Arrange observations

    Use rows for observations and columns for variables, then subtract each column’s mean.

  2. 2. Compute the SVD

    Factor the centered data matrix as X = UΣVᵀ with NumPy’s reduced SVD.

  3. 3. Square singular values

    Divide each squared singular value by n − 1 for the usual sample covariance.

  4. 4. Reconstruct and check

    Form V diag(s²/(n − 1)) Vᵀ and compare it with numpy.cov.

If your data matrix has observations in rows and variables in columns, the covariance matrix can be calculated directly from the singular values and right singular vectors of the centered data. The main things to check are the data orientation and the divisor: NumPy’s default covariance uses the sample divisor n−1n-1.

This article derives the formula, shows how to implement it with numpy.linalg.svd, and checks the result against numpy.cov.

The SVD covariance formula

Let AA be a real n×pn\times p data matrix, with nn observations and pp variables. Center each variable by subtracting its mean across observations:

Xij=Aij−1n∑r=1nArj.X_{ij}=A_{ij}-\frac{1}{n}\sum_{r=1}^{n}A_{rj}.

The sample covariance matrix is

C=1n−1XTX.C=\frac{1}{n-1}X^{\mathsf T}X.

Now take the singular value decomposition of the centered matrix:

X=UΣVT.X=U\Sigma V^{\mathsf T}.

Since UTU=IU^{\mathsf T}U=I, multiplying out gives

XTX=VΣTΣVT.X^{\mathsf T}X=V\Sigma^{\mathsf T}\Sigma V^{\mathsf T}.

The diagonal entries of ΣTΣ\Sigma^{\mathsf T}\Sigma are the squared singular values. If s is the vector of singular values and Vh is NumPy’s returned VTV^{\mathsf T}, then

C=V diag⁡ ⁣(s2n−1)VT.C=V\,\operatorname{diag}\!\left(\frac{s^2}{n-1}\right)V^{\mathsf T}.

This form works with NumPy’s reduced SVD, including when there are more variables than observations. In that case, Vh has only as many rows as there are singular values, but the reconstructed covariance still has shape p×pp\times p.

NumPy implementation

import numpy as np

A = np.array([
    [1.0, 2.0],
    [2.0, 4.0],
    [3.0, 6.0],
])

n = A.shape[0]
X = A - A.mean(axis=0)  # center each variable

U, s, Vh = np.linalg.svd(X, full_matrices=False)

# For real data this is V diag(s**2 / (n - 1)) V.T;
# the conjugation also matches np.cov for complex data.
cov_svd = (Vh.T * (s**2 / (n - 1))) @ Vh.conj()

# Independent check using NumPy's default sample covariance
cov_numpy = np.cov(A, rowvar=False)

print(cov_svd)
print(cov_numpy)
print(np.allclose(cov_svd, cov_numpy))

Both covariance calculations give, up to floating-point rounding,

[[1. 2.]
 [2. 4.]]
True

The centered matrix in this example is

X=[−1−20012].X=\begin{bmatrix}-1&-2\\0&0\\1&2\end{bmatrix}.

Its nonzero singular value is 10\sqrt{10}, and its right singular vector is (1,2)T/5(1,2)^{\mathsf T}/\sqrt{5}. The covariance is therefore

103−1[1/52/52/54/5]=[1224].\frac{10}{3-1}\begin{bmatrix}1/5&2/5\\2/5&4/5\end{bmatrix} =\begin{bmatrix}1&2\\2&4\end{bmatrix}.

The second singular value is zero because the two columns are perfectly dependent. A rank-deficient data set is not a problem: zero singular values contribute zero to the covariance.

Match NumPy’s conventions

By default, np.cov treats rows as variables and uses the sample divisor n−1n-1. With observations in rows, specify rowvar=False:

np.cov(A, rowvar=False)

For the population covariance, divide by nn instead. Equivalently, use np.cov(A, rowvar=False, bias=True) or ddof=0. If you use ddof=1, the default sample convention is explicit. With only one observation, sample covariance is undefined because its divisor is zero.

The SVD expression assumes you centered the data first. Applying it directly to uncentered data computes a second-moment matrix about zero, not the usual covariance around the sample means.

Common mistakes

  • Using the wrong orientation: If rows are observations, np.cov(A) interprets them as variables. Use rowvar=False when comparing results.
  • Forgetting to center: Compute the SVD of A - A.mean(axis=0), not generally of A itself.
  • Using the wrong divisor: Divide squared singular values by n−1n-1 to match NumPy’s default sample covariance; divide by nn for population covariance.
  • Reconstructing the wrong matrix: The feature covariance is XTX/(n−1)X^{\mathsf T}X/(n-1), so use the right singular vectors, Vh.T and Vh. The expression involving U instead describes the observation-space matrix XXT/(n−1)XX^{\mathsf T}/(n-1), which has a different shape.
  • Expecting exact equality: SVD and covariance routines can accumulate rounding error differently. Compare with np.allclose, not ==.

For complex-valued data, NumPy’s covariance convention conjugates the second variable. With observations in rows, the centered covariance is X.T @ X.conj() / (n - 1). If X = UΣVh, this gives the SVD expression Vh.T diag(s**2 / (n - 1)) Vh.conj(). The code above uses this form; for real-valued data it reduces to the transpose formula derived earlier. NumPy’s SVD returns Vh as the conjugate transpose of the right singular-vector matrix.

References

  1. NumPy documentation: numpy.linalg.svd
  2. NumPy documentation: numpy.cov

Spotted an error? Send a correction. Read how we write and check articles in our editorial policy.