SVD: The Fundamental Theorem of Data

Every Matrix Tells the Same Story

The Singular Value Decomposition says that every matrix $A \in \mathbb{R}^{m \times n}$ can be written as:

$$A = U \Sigma V^T$$

where $U$ is $m \times m$ orthogonal, $V$ is $n \times n$ orthogonal, and $\Sigma$ is diagonal with non-negative entries $\sigma_1 \geq \sigma_2 \geq \cdots \geq 0$.

In words: every linear transformation is a rotation, followed by a scaling along coordinate axes, followed by another rotation. No matter how tangled the matrix looks, its action decomposes into these three clean steps.

1. The Geometry: Ellipsoids From Spheres

Apply $A$ to the unit sphere $\{x : \|x\| = 1\}$ in $\mathbb{R}^n$. The image is an ellipsoid in $\mathbb{R}^m$.

• The directions of the ellipsoid’s axes are the left singular vectors (columns of $U$).
• The lengths of the semi-axes are the singular values $\sigma_i$.
• The orientations on the input sphere that map to these axes are the right singular vectors (columns of $V$).

The rank of $A$ is the number of nonzero singular values—the dimension of the ellipsoid. The condition number $\sigma_1 / \sigma_r$ measures its eccentricity: a large condition number means the ellipsoid is extremely elongated, and the linear system $Ax = b$ is sensitive to perturbations along the thin directions.

2. Eckart-Young: Optimal Compression

Theorem: The best rank-$k$ approximation to $A$ in both Frobenius and operator norm is:

$$A_k = \sum_{i=1}^{k} \sigma_i \, u_i v_i^T$$

“Best” means minimizing $\|A - B\|$ over all matrices $B$ with $\operatorname{rank}(B) \leq k$. The SVD solves this by keeping the $k$ largest singular values and discarding the rest.

Why this matters: The fraction of “energy” captured by rank $k$ is:

$$\frac{\sum_{i=1}^k \sigma_i^2}{\sum_{i=1}^r \sigma_i^2}$$

If the singular values decay rapidly, a few components capture most of the structure. This is not a heuristic—it is the provably optimal low-rank approximation.

Connection to projection: just as conditional expectation projects $Y$ onto the subspace generated by $X$ (keeping the “explained” component and discarding orthogonal noise), the truncated SVD projects the data matrix onto its $k$-dimensional principal subspace.

3. PCA Is SVD in Disguise

Given a data matrix $X \in \mathbb{R}^{n \times d}$ (rows = samples, columns = features), centered so each column has mean zero. The sample covariance is $C = \frac{1}{n-1}X^TX$.

The eigendecomposition of $C = V \Lambda V^T$ gives the principal components. But $C = \frac{1}{n-1}(X^TX)$, and if $X = U\Sigma V^T$, then $X^TX = V\Sigma^2 V^T$.

So the right singular vectors of $X$ are the principal components, and the singular values of $X$ determine the explained variance: $\lambda_i = \sigma_i^2/(n-1)$.

In practice, you never form the $d \times d$ covariance matrix. You compute the SVD of $X$ directly—which is numerically more stable and works even when $d > n$.

4. The Four Fundamental Subspaces

The SVD reveals the complete anatomy of a linear map $A: \mathbb{R}^n \to \mathbb{R}^m$.

Let $r = \operatorname{rank}(A)$. The singular vectors split the domain and codomain into orthogonal complements:

$$\mathbb{R}^n = \underbrace{\operatorname{Row}(A)}_{r\text{-dim}} \oplus \underbrace{\operatorname{Null}(A)}_{(n-r)\text{-dim}}$$ $$\mathbb{R}^m = \underbrace{\operatorname{Col}(A)}_{r\text{-dim}} \oplus \underbrace{\operatorname{Null}(A^T)}_{(m-r)\text{-dim}}$$

$A$ is an isomorphism from Row$(A)$ to Col$(A)$—it maps $v_i$ to $\sigma_i u_i$. On Null$(A)$, it annihilates. On Null$(A^T)$, nothing lands. The SVD makes this perfectly explicit: $V$ provides the basis for domain, $U$ for codomain, and $\Sigma$ governs the stretching between them.

5. Pseudoinverse and Least Squares

When $Ax = b$ has no solution (overdetermined system), the least squares solution minimizes $\|Ax - b\|^2$. The SVD gives it directly:

$$x^+ = A^+ b = V \Sigma^+ U^T b$$

where $\Sigma^+$ inverts the nonzero singular values and zeros out the rest. This is the Moore-Penrose pseudoinverse.

The geometry: project $b$ onto Col$(A)$ (via $UU^T$), then invert the map on the row space. If $b$ has components in Null$(A^T)$, those are silently discarded. If the system is underdetermined, the pseudoinverse chooses the minimum-norm solution.

The pseudoinverse is the SVD’s answer to the question: “What is the best you can do with an imperfect linear system?”

Why SVD Is Universal

The SVD is not one algorithm—it is the structural theorem behind:

Application	What SVD reveals
PCA	Principal directions of variance
Least Squares	Minimum-norm projection
Low-rank approximation	Optimal compression (Eckart-Young)
Condition number	Sensitivity to perturbation
Pseudoinverse	Best “undo” for a non-invertible map
Matrix norms	$\|A\|_2 = \sigma_1$, $\|A\|_F = \sqrt{\sum \sigma_i^2}$

Any time you have a matrix and need to understand its geometry—what it stretches, what it kills, and what subspace captures the action—the SVD is the answer.