Skip to main content

Eigenvalues and Eigenvectors Reveal Invariant Action

What This Concept Is​

Most vectors are rotated, sheared, or otherwise distorted by a matrix $A$. Eigenvectors are the exceptional directions that survive $A$ with only a rescaling: there exists a scalar $\lambda$ such that

$$A v = \lambda v, \quad v \ne 0.$$

The scalar $\lambda$ is the eigenvalue; $v$ is its eigenvector. The pair $(\lambda, v)$ is an invariant direction for the action of $A$. Geometrically, the line through $v$ is mapped to itself by $A$.

Finding eigenvalues uses the characteristic polynomial: $\lambda$ is an eigenvalue iff $(A - \lambda I)$ is singular, i.e., $\det(A - \lambda I) = 0$. This yields an $n$-th degree polynomial in $\lambda$ whose roots are the eigenvalues. For each root, the eigenspace $E_\lambda = \text{Null}(A - \lambda I)$ contains all eigenvectors for that eigenvalue together with the zero vector; its dimension is the geometric multiplicity. The algebraic multiplicity is the root's multiplicity in the characteristic polynomial; always $\text{geom mult} \le \text{alg mult}$, and equality characterizes diagonalizability (Cluster 4 concept 12).

Distinguish eigenvalues from related spectral quantities:

  • Eigenvalue $\lambda$: can be complex, can be zero, can be negative; ties to characteristic polynomial.
  • Singular value $\sigma_i$: always real and $\ge 0$; square roots of eigenvalues of $A^TA$; defined for all matrices including rectangular ones.
  • Spectral radius $\rho(A)$: the largest $|\lambda|$; governs long-run growth of $|A^k|$.

Two traces-and-dets identities worth burning in:

$$\text{tr}(A) = \sum_i \lambda_i, \qquad \det(A) = \prod_i \lambda_i.$$

These are free consistency checks and the cheapest way to catch arithmetic mistakes.

This is the first serious form of spectral thinking: instead of tracking all coordinates directly, look for invariant directions and their growth, decay, or sign flip. Once you pivot to this viewpoint, repeated-action problems become bookkeeping on scalar growth rates rather than on full matrix products.

Why It Matters Here​

Eigen-analysis explains: repeated matrix powers $A^k$ and their long-run behavior; stability of discrete-time dynamical systems ($|\lambda| < 1$ means decay, $|\lambda| > 1$ means growth); dominant directions in iterative processes (the largest-$|\lambda|$ eigenvector is what power iteration converges to); graph ranking and network flow behavior; positive-definite and SVD structure in Cluster 5.

Many CS applications care more about dominant modes than full coordinate detail. Eigenvectors provide those modes. PageRank, spectral clustering, vibration analysis, Google's search ranking, quantum state evolution, and the stability of numerical integrators all reduce to "find the right eigenvalues."

Concrete Examples​

Example 1 -- diagonal matrix, eigenvectors are standard basis. For

$$A = \begin{pmatrix} 3 & 0 \ 0 & 1 \end{pmatrix},$$

the standard basis vectors are eigenvectors: $A e_1 = 3 e_1$ and $A e_2 = e_2$. Eigenvalues are $3$ and $1$. Repeated application stretches the $e_1$-direction $3^k$ times faster than the $e_2$-direction: $A^k (a e_1 + b e_2) = 3^k a,e_1 + b,e_2$. For large $k$, the $e_1$ component dominates unless the initial vector has no component there.

Example 2 -- symmetric $2\times 2$. For

$$A = \begin{pmatrix} 2 & 1 \ 1 & 2 \end{pmatrix},$$

the characteristic polynomial is $(2-\lambda)^2 - 1 = \lambda^2 - 4\lambda + 3 = (\lambda - 1)(\lambda - 3)$. Eigenvalues are $\lambda_1 = 3$ and $\lambda_2 = 1$. Solve $(A - 3I)v = 0$: $-v_1 + v_2 = 0$, so $v_1 = (1, 1)^T$. Solve $(A - I)v = 0$: $v_1 + v_2 = 0$, so $v_2 = (1, -1)^T$. Trace check: $1 + 3 = 4 = 2 + 2$. Determinant check: $1 \cdot 3 = 3 = 4 - 1$. Under repeated multiplication, the $(1,1)$ direction grows $3^k$; the $(1,-1)$ direction stays at $1^k = 1$. After many iterations, any starting vector aligns with $(1, 1)$.

Example 3 -- defective matrix with repeated eigenvalue. For

$$A = \begin{pmatrix} 4 & 1 \ 0 & 4 \end{pmatrix},$$

the characteristic polynomial is $(\lambda - 4)^2$, so $\lambda = 4$ with algebraic multiplicity 2. But solving $(A - 4I)v = 0$ gives only $v = (1, 0)^T$ (and scalar multiples). Geometric multiplicity is 1, less than algebraic 2. $A$ is defective -- no basis of eigenvectors exists. This is the Jordan-block world (covered in Appendix B of Strang); such matrices cannot be diagonalized, though they can be put in Jordan form.

Common Confusion / Misconceptions​

"Eigenvectors are the basis vectors after diagonalization." Not exactly. They are specific invariant directions of the original transformation. If enough independent ones exist, they form a basis and diagonalize the matrix; if not (defective case), no diagonalizing basis exists.

"Find eigenvectors by guessing from a picture." Unreliable beyond $2 \times 2$. Always solve $(A - \lambda I)v = 0$ after finding $\lambda$ from the characteristic polynomial (or via np.linalg.eig).

"Eigenvalues are always real." Only guaranteed for symmetric (or Hermitian) matrices. A rotation matrix in $\mathbb{R}^2$ has complex eigenvalues $e^{\pm i\theta}$ -- real eigenvectors would contradict rotation's lack of invariant real directions (except multiples of rotation axes in 3D).

"Eigenvalue zero means the matrix is broken." It means the matrix is singular (has a nontrivial nullspace). This is structurally important but not pathological -- it happens whenever the transformation collapses at least one direction to zero, e.g., a projection onto a subspace.

"Eigenvalues equal singular values." True only for symmetric positive semidefinite matrices. For a general matrix, singular values equal $\sqrt{\text{eig}(A^TA)}$, and the two sets can be completely different.

How To Use It​

To find eigenvalues and eigenvectors:

  1. Write the characteristic polynomial $p(\lambda) = \det(A - \lambda I)$ and find its roots. For $n \ge 5$, this is numerically delicate; use library routines.
  2. For each eigenvalue $\lambda$, solve $(A - \lambda I)v = 0$ to obtain the eigenspace basis.
  3. Classify: simple (alg mult $= 1$), semisimple (alg $=$ geom), defective (alg $>$ geom).
  4. Interpret what each eigenvalue does:
    • $|\lambda| > 1$: expansion under iteration;
    • $|\lambda| < 1$: decay;
    • $\lambda < 0$: sign flip;
    • $\lambda$ complex: rotation with scaling;
    • $\lambda = 0$: collapse along that direction.
  5. Always connect the result back to repeated action or stability before claiming you understand.

In code: vals, vecs = np.linalg.eig(A) for a general matrix, np.linalg.eigh(A) for symmetric (faster, always real, returns orthonormal eigenvectors). For very large sparse matrices, use scipy.sparse.linalg.eigs (Krylov methods).

Transfer / Where This Shows Up Later​

  • S2 (algorithms): Spectral graph theory. Eigenvalues of the Laplacian count connected components (zero eigenvalue) and give the Cheeger bound on graph conductance (second smallest).
  • S4 (computer organization): Numerical stability of iterative solvers depends on the spectrum of the iteration matrix; dominant eigenvalue determines convergence rate.
  • S5 (queueing & Markov chains): Stationary distribution is the left eigenvector of the transition matrix for eigenvalue 1; mixing time is governed by the second-largest-$|\lambda|$.
  • S7 (architecture): Dependency-graph spectral analysis can reveal tightly coupled components (clusters of high-magnitude eigenvalues).
  • S8 (ranking): PageRank is exactly the dominant eigenvector of the teleporting random-walk matrix. Spectral clustering uses the eigenvectors of a normalized Laplacian.
  • Phase 7 (ML): PCA principal components are eigenvectors of the covariance matrix. Gradient-descent convergence rate near a minimum depends on the Hessian's eigenvalues. Gaussian process kernels are analyzed through their spectra. Vanishing/exploding gradients in RNNs are about spectral radii of weight matrices.

Check Yourself​

  1. Why are eigenvectors important for repeated powers of $A$?
  2. What does an eigenvalue of $0$ tell you?
  3. Why can a matrix have repeated eigenvalues but too few eigenvectors (defective)?
  4. Why is $\text{tr}(A) = \sum \lambda_i$ and $\det(A) = \prod \lambda_i$, and how can you use these as cheap consistency checks?
  5. Why are symmetric-matrix eigenvalues always real?
  6. How does a complex eigenvalue pair $a \pm bi$ show up geometrically in $\mathbb{R}^2$?

Mini Drill or Application​

  1. Find eigenvalues and eigenvectors of $\begin{pmatrix} 2 & 1 \ 1 & 2 \end{pmatrix}$. Verify trace and determinant checks.
  2. Decide which direction dominates under repeated multiplication by this matrix. Compute $A^{10} v$ for $v = (1, 0)^T$ numerically and confirm it aligns with the dominant eigenvector.
  3. In NumPy: A = np.array([[2,1],[1,2]]); w, V = np.linalg.eig(A); print(w); print(V). Verify $A V = V \text{diag}(w)$.
  4. Find the eigenvalues of the rotation matrix $R_{\pi/3} = \begin{pmatrix} 1/2 & -\sqrt{3}/2 \ \sqrt{3}/2 & 1/2 \end{pmatrix}$. Confirm they are complex conjugates on the unit circle. What real-valued invariant subspace survives rotation?
  5. Construct a defective $2\times 2$ matrix of your choice. Confirm its eigenvalue decomposition fails by trying np.linalg.eig; examine what the returned eigenvectors look like.

Read This Only If Stuck​