Positive Semi-Definite Matrix

• categories: Linear algebra, Matrix

Definition
A square Matrix $A\in {\mathbb{R}}^{n\times n}$ (or ${\mathbb{C}}^{n\times n}$) is positive semi-definite (PSD) if:

1. $A$ is Symmetric Matrix (or Hermitian if $A$ is complex): $A=A^T$ (or $A=A^{\ast }$).
2. For all vectors $x\in {\mathbb{R}}^n$ (or ${\mathbb{C}}^n$):
   $x^{T}Ax\ge 0.$

Intuition
A positive semi-definite matrix generalizes the concept of a Positive Definite Matrix by allowing the quadratic form $x^{T}Ax$ to be zero for some non-zero $x$. Geometrically, this can correspond to flat directions in a quadratic function, indicating that the function is not strictly convex but still non-negative.

Key Properties

1. Eigenvalues:
   All eigenvalues of $A$ are non-negative: ${\lambda }_i\ge 0$.
2. Determinant and Leading Principal Minors:
   • $\det (A)\ge 0$.
   • All leading principal minors are non-negative.
3. Cholesky Decomposition:
   If $A$ is positive semi-definite, it can be factorized as:
   $A=L{L}^T,$
   where $L$ is a lower triangular matrix, though $L$ may not be unique.
4. Relation to Positive Definiteness:
   $A$ is PSD if and only if it is positive definite or singular (non-invertible).
5. Norm Property:
   $\Vert Ax{\Vert }_2^2\ge 0\text{for all }x.$

Applications

1. Optimization:
   Positive semi-definite Hessian matrix indicate that the function is Convex Function (but not necessarily strictly convex).
2. Machine Learning:
   PSD matrices often represent covariance matrices in Gaussian processes, kernel methods, and other probabilistic models.