Skip to main content

Positive Definite Matrices Turn Quadratic Forms into Geometry

What This Concept Is​

A symmetric real matrix $A$ is positive definite (PD) iff

$$x^T A x > 0 \quad \text{for every nonzero } x \in \mathbb{R}^n,$$

and positive semidefinite (PSD) iff $x^T A x \ge 0$. Symmetry is part of the definition (the standard theory needs it for the spectral picture to work cleanly); an asymmetric matrix satisfying the inequality is usually turned into $\tfrac{1}{2}(A + A^T)$ first.

Positive definiteness has five equivalent characterizations (any one of them proves all the others):

  1. $x^T A x > 0$ for all nonzero $x$ (the definition).
  2. All eigenvalues of $A$ are strictly positive.
  3. All leading principal minors $\det(A_k)$ (top-left $k \times k$ submatrix determinants) are strictly positive -- Sylvester's criterion.
  4. There exists an invertible $R$ such that $A = R^T R$; the Cholesky decomposition $A = L L^T$ with lower-triangular $L$ having positive diagonal is the canonical one.
  5. All pivots in Gaussian elimination (without row swaps) are positive.

Distinguish these related conditions; they do not form a single strength ordering:

  • Positive definite: $x^T A x > 0$, $\lambda_i > 0$.
  • Positive semidefinite: $x^T A x \ge 0$, $\lambda_i \ge 0$ (nullspace allowed).
  • Indefinite: some positive and some negative eigenvalues (saddle behavior).
  • Negative semidefinite / negative definite: flipped signs.
  • Symmetric: just $A = A^T$, no sign condition.

The quadratic form $Q(x) = x^T A x$ defines a function from $\mathbb{R}^n$ to $\mathbb{R}$. If $A$ is PD, $Q$ looks like a bowl centered at the origin: level sets ${x : Q(x) = c}$ are ellipsoids whose axis lengths are $\sqrt{c/\lambda_i}$ along the corresponding eigenvector directions. If indefinite, level sets are hyperboloids (saddles). Geometry, algebra, and optimization align cleanly when $A$ is PD.

Why It Matters Here​

Positive definiteness explains: why least-squares normal equations $A^TA \hat x = A^T b$ have a unique solution when $A$ has independent columns (because $A^TA$ is PD); why some quadratic objectives have unique minima (PD Hessian); why Cholesky factorization exists and is stable (PD is the precondition); and how curvature enters optimization and ML.

This is the bridge from linear algebra into optimization thinking. Whenever a loss function is quadratic or locally quadratic near a minimum, a positive-definite Hessian at a stationary point is sufficient for a strict local minimum, not necessary: the scalar function x^4 has a strict minimum with zero second derivative there. Whenever you see a covariance matrix, it is PSD by construction; sample covariance is strictly PD when the centered data columns are independent (with at least two observations). Centering can remove a constant column. Dense Cholesky uses about half the leading factorization work of dense LU under the standard arithmetic model. Sparse structure and hardware still affect method choice.

Concrete Examples​

Example 1 -- diagonal PD. Take

$$A = \begin{pmatrix} 2 & 0 \ 0 & 5 \end{pmatrix}.$$

Then $x^T A x = 2 x_1^2 + 5 x_2^2 > 0$ for all nonzero $x$. Eigenvalues are 2 and 5 (positive). Level sets $2 x_1^2 + 5 x_2^2 = 1$ are ellipses with semi-axes $1/\sqrt{2}$ and $1/\sqrt{5}$ along the standard directions. PD.

Example 2 -- symmetric PD via eigenvalues. For

$$A = \begin{pmatrix} 2 & -1 \ -1 & 2 \end{pmatrix},$$

the characteristic polynomial is $\lambda^2 - 4\lambda + 3 = (\lambda-1)(\lambda-3)$. Eigenvalues $1, 3 > 0$, so PD. Sylvester's criterion: leading minors are $\det(A_1) = 2 > 0$ and $\det(A_2) = \det A = 3 > 0$. Cholesky: $A = L L^T$ with $L = \begin{pmatrix} \sqrt{2} & 0 \ -1/\sqrt{2} & \sqrt{3/2} \end{pmatrix}$ (verify by direct multiplication). All five characterizations agree.

Example 3 -- indefinite saddle. For

$$B = \begin{pmatrix} 1 & 0 \ 0 & -1 \end{pmatrix}, \quad x^T B x = x_1^2 - x_2^2.$$

Positive along the $x_1$ axis, negative along the $x_2$ axis, zero along the diagonal $x_1 = \pm x_2$. Level sets are hyperbolas. Eigenvalues $1, -1$ -- indefinite. Geometrically, this is a saddle, not a bowl. In optimization, a Hessian with this sign pattern at a critical point diagnoses a saddle rather than a min or max.

Example 4 -- PD by construction. For any real matrix $M$ with independent columns, $M^T M$ is SPD. Proof: $x^T M^T M x = |Mx|^2 \ge 0$; it is zero iff $Mx = 0$ iff $x = 0$ (column independence). This is why $A^T A$ in the normal equations is PD whenever the regression design matrix has independent features. This explains the normal-equation geometry. Forming the Gram matrix can worsen numerical sensitivity; compare QR or SVD before selecting a solver.

Common Confusion / Misconceptions​

"Positive diagonal entries alone make a matrix PD." False. Consider $A = \begin{pmatrix} 1 & 2 \ 2 & 1 \end{pmatrix}$: positive diagonal, but $x^T A x = x_1^2 + 4 x_1 x_2 + x_2^2$, which equals $-2$ at $x = (1, -1)^T$. Off-diagonal entries can destroy definiteness.

"Symmetric is the same as positive definite." Symmetry is a requirement of the standard theory but is not sufficient. $B = \begin{pmatrix} 1 & 0 \ 0 & -1 \end{pmatrix}$ is symmetric but not PD.

"PD means invertible." PD is strictly stronger. It implies invertibility (all eigenvalues nonzero), but the reverse is false: $-I$ is invertible but definitely not PD.

"Any matrix of the form $M^TM$ is positive definite." $M^T M$ is always PSD; it is PD iff $M$ has linearly independent columns. If $M$ has dependent columns (e.g., rank-deficient design matrix), $M^T M$ is only PSD and has a nontrivial nullspace -- one of the reasons regularization (ridge regression, adding $\lambda I$) is used.

"Sylvester's criterion uses all principal minors." No -- only the leading principal minors (top-left $k \times k$). For PSD you need to check all principal minors, not just leading ones.

How To Use It​

When checking positive definiteness:

  1. Confirm symmetry; if the matrix is not symmetric, work with $\tfrac{1}{2}(A + A^T)$ or step back to a broader theory.
  2. Use one of the five equivalent tests. For small hand work: Sylvester's criterion. For numerical work, attempt Cholesky with a stated pivot policy and verify reconstruction. Tiny-pivot behavior and success on rounded data do not by themselves prove exact definiteness of uncertain data.
  3. For exact PSD, use a valid criterion such as all principal minors nonnegative. Numerically, interpret tiny computed eigenvalues relative to error and scale rather than demanding a literal nonnegative bit pattern.
  4. Interpret the result in optimization language:
    • PD quadratic part: a unique global minimum for an unconstrained quadratic with any linear term.
    • PSD homogeneous quadratic: minimum zero, with possible flat directions. Adding a linear term can instead make a singular PSD quadratic unbounded below.
    • Indefinite: saddle behavior; not a minimum.
  5. Use Cholesky rather than LU whenever $A$ is known SPD; about half the cost.
  6. If $A$ is not PD but you need it to be (covariance regularization), a shift A+lambda*I is PD exactly when lambda>−lambda_min(A). A positive shift suffices for PSD input but not for every indefinite input. Report the changed model.

Transfer to the calibration project​

Use this concept to justify a result or reject an assumption in the calibration project. Name the matrix shape, objective, scalar field and available evidence before interpreting the result.

Check Yourself​

  1. Why is symmetry part of the standard definition of positive definiteness?
  2. What does $x^T A x$ represent geometrically when $A$ is PD?
  3. Why does a positive definite matrix have positive eigenvalues -- and vice versa?
  4. Why is $A^T A$ always PSD, and when is it strictly PD?
  5. What is the optimization meaning of the Hessian being indefinite at a critical point?
  6. What is Sylvester's criterion, and why does it fail for PSD without modification?

Mini Drill or Application​

For each matrix, classify it as positive definite, positive semidefinite, indefinite, or negative (semi)definite:

  1. $\begin{pmatrix} 3 & 0 \ 0 & 2 \end{pmatrix}$
  2. $\begin{pmatrix} 1 & 2 \ 2 & 1 \end{pmatrix}$
  3. $\begin{pmatrix} 2 & -1 \ -1 & 2 \end{pmatrix}$
  4. $\begin{pmatrix} 1 & 1 \ 1 & 1 \end{pmatrix}$

Then explain which one would define a strictly stable quadratic loss surface.

  1. In NumPy: generate X = np.random.randn(100, 5) and compute S = X.T @ X; verify S is SPD by attempting np.linalg.cholesky(S) (success = PD). Then generate a rank-deficient $X$ (repeat a column) and observe Cholesky failure.
  2. Derive the Cholesky factor $L$ of $\begin{pmatrix} 4 & 2 \ 2 & 5 \end{pmatrix}$ by hand. Verify $L L^T$ reproduces the matrix.

Read This Only If Stuck​

Use the bounded reading guide and the Strang book index. The book link opens an index; find the named topic in your edition, read a small section and reconstruct one argument with the book closed.

A positive pivot policy is not a theorem about noisy data​

For a symmetric exact matrix, leading principal minors being positive is equivalent to positive definiteness. Replacing strict positivity by nonnegativity does not give the corresponding PSD test using only leading minors: diag(0,-1) has leading determinants zero and zero but a negative direction.

Cholesky computes successive Schur-complement pivots. A floating implementation can reject a tiny positive pivot under its stated threshold; distinguish that from an exact negative-direction witness. Centering data can reduce rank, and regularization A+lambda*I changes both curvature and the original objective. Transfer: P13, P15 and P19.

After an independent attempt, use practice feedback and model contracts to check your reasoning.