Skip to main content

Singular Value Decomposition Is the Universal Factorization

What This Concept Is​

Every real matrix $A \in \mathbb{R}^{m \times n}$, square or rectangular, full-rank or rank-deficient, admits a singular value decomposition

$$A = U \Sigma V^T$$

where $U \in \mathbb{R}^{m \times m}$ and $V \in \mathbb{R}^{n \times n}$ are orthogonal matrices, and $\Sigma \in \mathbb{R}^{m \times n}$ is rectangular diagonal with nonnegative entries $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_r > 0 = \cdots = 0$. The nonnegative numbers $\sigma_i$ are the singular values; $r$ equals the rank of $A$. The columns of $U$ are the left singular vectors (an orthonormal basis of $\mathbb{R}^m$); the columns of $V$ are the right singular vectors (an orthonormal basis of $\mathbb{R}^n$).

SVD is the universal factorization because it exists for every matrix, it simultaneously orthogonalizes the input and output sides, and it makes the action of $A$ geometrically transparent: $A$ takes an orthonormal basis $V$ in the input, rescales each direction by the corresponding $\sigma_i$, and deposits the result along an orthonormal basis $U$ in the output. Every matrix is a rotate -> stretch -> rotate.

Key identities that make SVD the master tool:

  • Singular values squared equal eigenvalues of $A^T A$: $\sigma_i^2 = \lambda_i(A^T A)$.
  • Right singular vectors $v_i$ are eigenvectors of $A^T A$; left singular vectors $u_i$ are eigenvectors of $A A^T$.
  • The four fundamental subspaces read off directly: $\text{Col}(A) = \text{span}{u_1, \ldots, u_r}$; $\text{Null}(A^T) = \text{span}{u_{r+1}, \ldots, u_m}$; $\text{Row}(A) = \text{span}{v_1, \ldots, v_r}$; $\text{Null}(A) = \text{span}{v_{r+1}, \ldots, v_n}$.
  • Eckart-Young theorem: the best rank-$k$ approximation in the 2-norm or Frobenius norm is $A_k = \sum_{i=1}^k \sigma_i u_i v_i^T$.
  • $|A|_2 = \sigma_1$ (spectral norm); $|A|_F = \sqrt{\sum \sigma_i^2}$ (Frobenius norm); $\kappa_2(A) = \sigma_1/\sigma_r$ (condition number).

Distinguish SVD from its cousins:

  • Eigendecomposition $A = S\Lambda S^{-1}$: requires a full eigenbasis; fails for defective or rectangular $A$.
  • Spectral theorem $A = Q\Lambda Q^T$: requires $A$ symmetric; SVD generalizes to all matrices.
  • Full SVD uses $U \in \mathbb{R}^{m \times m}$, $V \in \mathbb{R}^{n \times n}$; thin SVD uses $U \in \mathbb{R}^{m \times r}$, $V \in \mathbb{R}^{n \times r}$ and drops zero singular values -- usually what you want in numerics.
  • SVD vs PCA: PCA of a centered data matrix $X$ is the SVD of $X$ (or equivalently the eigendecomposition of $\tfrac{1}{n} X^T X$). They are the same calculation in different language; see the misconception below.

Why It Matters Here​

SVD unifies: rank, orthogonal bases, low-rank approximation, conditioning, compression, and latent structure extraction. In CS and data work, this is the decomposition that explains why matrices can often be approximated well by far fewer directions than their raw size suggests -- and why the approximation quality is exactly the tail sum of squared singular values.

Operationally, SVD is the most robust single tool for answering "what does this matrix really do, and how big are its answers?" It underlies LSI / latent semantic analysis in information retrieval, PCA in dimensionality reduction, recommender systems (Netflix prize), model compression, inverse-problem regularization (truncated SVD), and the pseudoinverse $A^+ = V \Sigma^+ U^T$ that gives the minimum-norm least-squares solution to any $Ax = b$.

Concrete Examples​

Example 1 -- $2 \times 2$ SVD. Let

$$A = \begin{pmatrix} 3 & 0 \ 4 & 5 \end{pmatrix}.$$

Compute $A^T A = \begin{pmatrix} 25 & 20 \ 20 & 25 \end{pmatrix}$. Eigenvalues of $A^T A$: $\det \begin{pmatrix} 25-\lambda & 20 \ 20 & 25-\lambda \end{pmatrix} = (25-\lambda)^2 - 400$, so $\lambda = 5$ or $\lambda = 45$. Singular values: $\sigma_1 = \sqrt{45} = 3\sqrt{5}$, $\sigma_2 = \sqrt{5}$. Right singular vectors: $v_1 = (1, 1)^T/\sqrt{2}$ for $\lambda_1 = 45$; $v_2 = (1, -1)^T/\sqrt{2}$ for $\lambda_2 = 5$. Left singular vectors from $u_i = A v_i / \sigma_i$:

$$u_1 = \frac{1}{3\sqrt{5}} A \begin{pmatrix} 1/\sqrt{2} \ 1/\sqrt{2} \end{pmatrix} = \frac{1}{3\sqrt{5}} \cdot \frac{1}{\sqrt{2}} \begin{pmatrix} 3 \ 9 \end{pmatrix} = \frac{1}{\sqrt{10}}\begin{pmatrix} 1 \ 3 \end{pmatrix}.$$

Similarly $u_2 = \tfrac{1}{\sqrt{10}}(3, -1)^T$. Check $U\Sigma V^T = A$.

Example 2 -- rank-1 dominant structure. Let

$$A = \begin{pmatrix} 1 & 2 \ 2 & 4 \ 3 & 6 \end{pmatrix}.$$

Every column is a multiple of $(1, 2, 3)^T$ and every row is a multiple of $(1, 2)$. Rank is 1. SVD has exactly one nonzero singular value: $\sigma_1 = \sqrt{70}$ (from $A^T A = \begin{pmatrix} 14 & 28 \ 28 & 56 \end{pmatrix}$ with eigenvalue 70). $u_1 = (1, 2, 3)^T/\sqrt{14}$, $v_1 = (1, 2)^T/\sqrt{5}$. So $A = \sigma_1 u_1 v_1^T$ exactly -- one outer product reconstructs the whole matrix, which is the Eckart-Young theorem's cleanest instance.

Example 3 -- image compression intuition. A $512 \times 512$ grayscale image is a matrix with 262,144 entries. Typical natural images have singular values that decay rapidly after the first ~50. Keeping the top $k = 50$ SVD components stores $50 \cdot 512 + 50 + 50 \cdot 512 \approx 51{,}250$ numbers, a 5× compression with visually negligible loss. Truncate further for stronger compression at graceful degradation. This is the archetype of "low-rank approximation works because real data has concentrated singular-value spectrum."

Common Confusion / Misconceptions​

"Singular values are the eigenvalues of $A$." False in general. They are square roots of eigenvalues of $A^T A$. Only when $A$ is symmetric positive semidefinite do singular values coincide with eigenvalues; for a symmetric matrix with some negative eigenvalues, $\sigma_i = |\lambda_i|$.

"SVD = PCA." They are the same calculation applied to different objects. PCA diagonalizes the covariance matrix $C = \tfrac{1}{n-1} X_c^T X_c$ (where $X_c$ is the centered data matrix). SVD of $X_c$ gives singular values related to PCA eigenvalues by $\sigma_i^2 = (n-1)\lambda_i$. In practice, PCA is implemented via SVD of the centered data matrix because it avoids forming $X^T X$ explicitly.

"SVD is only a numerical-algorithms topic." Conceptually, it is one of the cleanest ways to understand what a matrix does to lengths and directions. The geometric rotate -> stretch -> rotate picture is the most general statement of "what is a linear map."

"Any nonzero singular value means a direction is 'important'." Relative magnitude matters. A matrix with $\sigma_1 = 10^6$ and $\sigma_2 = 10^{-6}$ is effectively rank 1 for most practical purposes; the second direction is not "important" in any downstream sense. This is the entire point of low-rank approximation.

How To Use It​

Use SVD when you care about:

  1. Best low-rank approximation.
  2. Dominant directions in input/output.
  3. Noise versus signal separation (truncate below a threshold).
  4. Pseudoinverse reasoning for rectangular or rank-deficient $A$.
  5. Rectangular matrices where eigenvectors are not the right language.
  6. Rank determination (number of singular values above a numerical tolerance).
  7. Condition number $\kappa_2 = \sigma_1/\sigma_r$ for stability assessment.

Interpret each singular triplet $(\sigma_i, u_i, v_i)$ as: $\sigma_i$ = magnitude of action along the $i$-th direction pair; $v_i$ = input direction; $u_i$ = output direction.

In code: U, s, Vt = np.linalg.svd(A, full_matrices=False) returns thin SVD; for very large sparse matrices, scipy.sparse.linalg.svds(A, k=50) returns just the top $k$. Condition number: np.linalg.cond(A). Pseudoinverse: np.linalg.pinv(A).

Transfer / Where This Shows Up Later​

  • S2 (algorithms): Randomized SVD gives $O(mn \log k)$ approximation algorithms for the top-$k$ components, used for massive matrices.
  • S4 (computer organization): Truncated SVD on weight matrices is a standard model-compression technique for deploying ML models on-device.
  • S5 (queueing & signal processing): Principal component analysis of signal covariance is SVD of the data matrix; used in noise reduction and spectral estimation.
  • S7 (architecture): Interaction matrices between services can be summarized by their top singular values, exposing dominant coupling modes.
  • S8 (ranking & recommendations): Latent semantic indexing, the original Netflix Prize approach, and modern matrix-factorization recommenders are all truncated SVDs (or regularized cousins) on user-item matrices. Word embeddings like GloVe are SVDs of log-co-occurrence matrices.
  • Phase 7 (ML): PCA is SVD of the centered data matrix. LoRA (low-rank adaptation of large language models) constrains fine-tuning updates to be of the form $\Delta W = B A$ with low rank -- a structural SVD-like restriction. Autoencoder bottlenecks learn a nonlinear analog of truncated SVD. Whitening preprocesses data with $\Sigma^{-1/2}$ computed via SVD of the covariance.

Check Yourself​

  1. Why is SVD available for every matrix?
  2. What do small singular values tell you about information loss?
  3. Why is truncated SVD a principled compression method? What is the error bound?
  4. Why are $A^TA$ eigenvectors equal to right singular vectors?
  5. What is the relationship between SVD and the four fundamental subspaces?
  6. How does $\kappa_2(A) = \sigma_1/\sigma_r$ relate to the geometry of the ellipsoid image of the unit ball?

Mini Drill or Application​

  1. Explain in words how SVD would compress an image matrix. If the top 10 singular values hold 95% of the squared Frobenius norm, what rank approximation preserves that energy?
  2. If a matrix has rank 2, how many nonzero singular values can it have? Why?
  3. Describe what the largest singular value tells you operationally about $|Ax|/|x|$.
  4. In NumPy: U, s, Vt = np.linalg.svd(A, full_matrices=False). Reconstruct $A$ via A_rec = U @ np.diag(s) @ Vt and check closeness. Truncate to top $k$: A_k = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]. Compute the Frobenius-norm error and compare to the theoretical value $\sqrt{\sum_{i>k} \sigma_i^2}$.
  5. Compute the SVD of $A = \begin{pmatrix} 1 & 0 \ 1 & 1 \ 0 & 1 \end{pmatrix}$ by finding eigenvalues of $A^T A = \begin{pmatrix} 2 & 1 \ 1 & 2 \end{pmatrix}$. Write the full $A = U\Sigma V^T$ by hand and verify.
  6. Show that for any matrix $A$, $|A|_F^2 = \sum_i \sigma_i^2$ and $|A|_2 = \sigma_1$ directly from the SVD.

Read This Only If Stuck​