Skip to main content

Singular Value Decomposition Is the Universal Factorization

What This Concept Is​

Every real matrix $A \in \mathbb{R}^{m \times n}$, square or rectangular, full-rank or rank-deficient, admits a singular value decomposition

$$A = U \Sigma V^T$$

where $U \in \mathbb{R}^{m \times m}$ and $V \in \mathbb{R}^{n \times n}$ are orthogonal matrices, and $\Sigma \in \mathbb{R}^{m \times n}$ is rectangular diagonal with nonnegative entries $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_r > 0 = \cdots = 0$. The nonnegative numbers $\sigma_i$ are the singular values; $r$ equals the rank of $A$. The columns of $U$ are the left singular vectors (an orthonormal basis of $\mathbb{R}^m$); the columns of $V$ are the right singular vectors (an orthonormal basis of $\mathbb{R}^n$).

SVD is the universal factorization because it exists for every matrix, it simultaneously orthogonalizes the input and output sides, and it makes the action of $A$ geometrically transparent: $A$ takes an orthonormal basis $V$ in the input, rescales each direction by the corresponding $\sigma_i$, and deposits the result along an orthonormal basis $U$ in the output. The orthogonal factors can include reflections; the rectangular diagonal factor can embed or discard dimensions as well as scale them.

Key identities that make SVD the master tool:

  • Singular values squared equal eigenvalues of $A^T A$: $\sigma_i^2 = \lambda_i(A^T A)$.
  • Right singular vectors $v_i$ are eigenvectors of $A^T A$; left singular vectors $u_i$ are eigenvectors of $A A^T$.
  • The four fundamental subspaces read off directly: $\text{Col}(A) = \text{span}{u_1, \ldots, u_r}$; $\text{Null}(A^T) = \text{span}{u_{r+1}, \ldots, u_m}$; $\text{Row}(A) = \text{span}{v_1, \ldots, v_r}$; $\text{Null}(A) = \text{span}{v_{r+1}, \ldots, v_n}$.
  • Eckart-Young theorem: a best approximation of rank at most k in the 2-norm or Frobenius norm is $A_k = \sum_{i=1}^k \sigma_i u_i v_i^T$.
  • $|A|_2 = \sigma_1$ (spectral norm); $|A|_F = \sqrt{\sum \sigma_i^2}$ (Frobenius norm); for a nonsingular square matrix, $\kappa_2(A)=\sigma_1/\sigma_n$. A singular square matrix has infinite solve condition number; the ratio of largest to smallest positive singular value instead describes a restricted nonzero subspace.

Distinguish SVD from its cousins:

  • Eigendecomposition $A = S\Lambda S^{-1}$: requires a full eigenbasis; fails for defective or rectangular $A$.
  • Spectral theorem $A = Q\Lambda Q^T$: requires $A$ symmetric; SVD generalizes to all matrices.
  • Full SVD uses $U \in \mathbb{R}^{m \times m}$, $V \in \mathbb{R}^{n \times n}$; reduced/economy SVD commonly retains p=min(m,n) singular values, including zeros. A compact rank-r SVD drops the zero terms. State dimensions because terminology varies.
  • SVD vs PCA: PCA of a centered data matrix $X$ is the SVD of $X$ (or equivalently the eigendecomposition of $\tfrac{1}{n} X^T X$). They are the same calculation in different language; see the misconception below.

Why It Matters Here​

SVD unifies: rank, orthogonal bases, low-rank approximation, conditioning, compression, and latent structure extraction. In CS and data work, this is the decomposition that explains why matrices can often be approximated well by far fewer directions than their raw size suggests -- and why the squared Frobenius error of truncated SVD is the tail sum of squared singular values, while its operator 2-norm error is the next singular value.

Operationally, SVD is the most robust single tool for answering "what does this matrix really do, and how big are its answers?" It underlies LSI / latent semantic analysis in information retrieval, PCA in dimensionality reduction, recommender systems (Netflix prize), model compression, inverse-problem regularization (truncated SVD), and the pseudoinverse $A^+ = V \Sigma^+ U^T$ that gives the minimum-norm least-squares solution to any $Ax = b$.

Concrete Examples​

Example 1 -- $2 \times 2$ SVD. Let

$$A = \begin{pmatrix} 3 & 0 \ 4 & 5 \end{pmatrix}.$$

Compute $A^T A = \begin{pmatrix} 25 & 20 \ 20 & 25 \end{pmatrix}$. Eigenvalues of $A^T A$: $\det \begin{pmatrix} 25-\lambda & 20 \ 20 & 25-\lambda \end{pmatrix} = (25-\lambda)^2 - 400$, so $\lambda = 5$ or $\lambda = 45$. Singular values: $\sigma_1 = \sqrt{45} = 3\sqrt{5}$, $\sigma_2 = \sqrt{5}$. Right singular vectors: $v_1 = (1, 1)^T/\sqrt{2}$ for $\lambda_1 = 45$; $v_2 = (1, -1)^T/\sqrt{2}$ for $\lambda_2 = 5$. Left singular vectors from $u_i = A v_i / \sigma_i$:

$$u_1 = \frac{1}{3\sqrt{5}} A \begin{pmatrix} 1/\sqrt{2} \ 1/\sqrt{2} \end{pmatrix} = \frac{1}{3\sqrt{5}} \cdot \frac{1}{\sqrt{2}} \begin{pmatrix} 3 \ 9 \end{pmatrix} = \frac{1}{\sqrt{10}}\begin{pmatrix} 1 \ 3 \end{pmatrix}.$$

Similarly $u_2 = \tfrac{1}{\sqrt{10}}(3, -1)^T$. Check $U\Sigma V^T = A$.

Example 2 -- rank-1 dominant structure. Let

$$A = \begin{pmatrix} 1 & 2 \ 2 & 4 \ 3 & 6 \end{pmatrix}.$$

Every column is a multiple of $(1, 2, 3)^T$ and every row is a multiple of $(1, 2)$. Rank is 1. SVD has exactly one nonzero singular value: $\sigma_1 = \sqrt{70}$ (from $A^T A = \begin{pmatrix} 14 & 28 \ 28 & 56 \end{pmatrix}$ with eigenvalue 70). $u_1 = (1, 2, 3)^T/\sqrt{14}$, $v_1 = (1, 2)^T/\sqrt{5}$. So $A = \sigma_1 u_1 v_1^T$ exactly -- one outer product reconstructs the whole matrix, which is the Eckart-Young theorem's cleanest instance.

Example 3 -- storage is not quality. A 512-by-512 matrix has 262,144 entries. Storing 50 left/right singular vectors and values uses 51,250 scalar entries. The byte ratio also depends on precision and format; the error depends on this particular image’s spectrum. Measure the squared tail energy and a task-relevant quality criterion before deciding that the loss is acceptable. There is no universal rank-50 visual guarantee.

Common Confusion / Misconceptions​

"Singular values are the eigenvalues of $A$." False in general. They are square roots of eigenvalues of $A^T A$. Only when $A$ is symmetric positive semidefinite do singular values coincide with eigenvalues; for a symmetric matrix with some negative eigenvalues, $\sigma_i = |\lambda_i|$.

"SVD = PCA." They are the same calculation applied to different objects. PCA diagonalizes the covariance matrix $C = \tfrac{1}{n-1} X_c^T X_c$ (where $X_c$ is the centered data matrix). SVD of $X_c$ gives singular values related to PCA eigenvalues by $\sigma_i^2 = (n-1)\lambda_i$. In practice, PCA is implemented via SVD of the centered data matrix because it avoids forming $X^T X$ explicitly.

"SVD is only a numerical-algorithms topic." Conceptually, it is one of the cleanest ways to understand what a matrix does to lengths and directions. The geometric rotate -> stretch -> rotate picture is the most general statement of "what is a linear map."

"A small singular value is irrelevant to every task." Importance depends on units, noise and the downstream quantity. A weakly represented direction can carry the signal of interest, while inverting it can amplify noise. State the truncation threshold, report discarded energy and test the effect on the intended decision.

How To Use It​

Use SVD when you care about:

  1. Best low-rank approximation.
  2. Dominant directions in input/output.
  3. Noise versus signal separation (truncate below a threshold).
  4. Pseudoinverse reasoning for rectangular or rank-deficient $A$.
  5. Rectangular matrices where eigenvectors are not the right language.
  6. Rank determination (number of singular values above a numerical tolerance).
  7. For nonsingular square A, condition number kappa_2=sigma_max/sigma_min; singular square A has infinite solve condition number.

Interpret each singular triplet $(\sigma_i, u_i, v_i)$ as: $\sigma_i$ = magnitude of action along the $i$-th direction pair; $v_i$ = input direction; $u_i$ = output direction.

A reduced SVD commonly retains min(m,n) values; a truncated solver returns a chosen subset. Check factor shapes, ordering, convergence and reconstruction in the actual API. A numerical pseudoinverse also needs a cutoff. The offline lab uses supplied analytic factors so no external SVD library is required.

Transfer to the calibration project​

Use this concept to justify a result or reject an assumption in the calibration project. Name the matrix shape, objective, scalar field and available evidence before interpreting the result.

Check Yourself​

  1. Why is SVD available for every matrix?
  2. What do small singular values tell you about information loss?
  3. Why is truncated SVD a principled compression method? What is the error bound?
  4. Why are $A^TA$ eigenvectors equal to right singular vectors?
  5. What is the relationship between SVD and the four fundamental subspaces?
  6. How does $\kappa_2(A) = \sigma_1/\sigma_r$ relate to the geometry of the ellipsoid image of the unit ball?

Mini Drill or Application​

  1. Explain in words how SVD would compress an image matrix. If the top 10 singular values hold 95% of the squared Frobenius norm, what rank approximation preserves that energy?
  2. If a matrix has rank 2, how many nonzero singular values can it have? Why?
  3. Describe what the largest singular value tells you operationally about $|Ax|/|x|$.
  4. In NumPy: U, s, Vt = np.linalg.svd(A, full_matrices=False). Reconstruct $A$ via A_rec = U @ np.diag(s) @ Vt and check closeness. Truncate to top $k$: A_k = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]. Compute the Frobenius-norm error and compare to the theoretical value $\sqrt{\sum_{i>k} \sigma_i^2}$.
  5. Compute the SVD of $A = \begin{pmatrix} 1 & 0 \ 1 & 1 \ 0 & 1 \end{pmatrix}$ by finding eigenvalues of $A^T A = \begin{pmatrix} 2 & 1 \ 1 & 2 \end{pmatrix}$. Write the full $A = U\Sigma V^T$ by hand and verify.
  6. Show that for any matrix $A$, $|A|_F^2 = \sum_i \sigma_i^2$ and $|A|_2 = \sigma_1$ directly from the SVD.

Read This Only If Stuck​

Use the bounded reading guide and the Strang book index. The book link opens an index; find the named topic in your edition, read a small section and reconstruct one argument with the book closed.

Report both approximation errors and what was discarded​

For diag(4,1), a rank-one truncation retains diag(4,0). The operator error is one and the Frobenius error is also one; for diag(4,2,1), those errors are two and sqrt(5). Do not report the squared tail as the norm itself.

Repeated singular values allow different optimal bases and sometimes different optimal approximations. Validate reconstruction, orthonormality and error, not the sign of a particular returned vector. Center PCA data explicitly and divide squared singular values by the chosen covariance denominator. Raw SVD and centered PCA answer different questions. Transfer: P14–P17.

After an independent attempt, use practice feedback and model contracts to check your reasoning.