- categories: Linear algebra, Matrix
Definition
A square Matrix (or ) is positive definite if:
- is symmetric (or Hermitian if is complex): (or ).
- For all non-zero vectors (or ):
(In the complex case, , where is the conjugate transpose of .)
Intuition
- A positive definite matrix corresponds to a quadratic form that always produces positive values for non-zero .
- It represents a multidimensional generalization of a convex parabola, where the energy (or quadratic form) reaches a unique minimum, often associated with optimization and stability.
In Deep Learning, a positive definite Hessian matrix (second-derivative matrix) of a loss function guarantees that the function is convex, ensuring the existence of a unique minimum.
Key Properties
- Eigenvalues:
All eigenvalues of are positive: . - Determinant and Leading Principal Minors:
- .
- All leading principal minors (determinants of top-left submatrices) are positive.
- Cholesky Decomposition:
A positive definite matrix can be decomposed as:
where is a lower triangular matrix with positive diagonal entries. - Matrix Inversion:
A positive definite matrix is always invertible. - Norm Property:
- Relation to Convexity:
- If is the Hessian of a scalar function , i.e., , then is strictly convex.
Applications in Deep Learning
- Optimization:
- Positive definite matrices arise in the Hessian of loss functions. If the Hessian is positive definite, the loss is convex, and gradient descent converges to the unique minimum.
- Convexity ensures no local minima exist, making optimization simpler and more predictable.
- Energy Landscape:
In machine learning models, ensuring a positive definite structure means the “energy” (loss function) has a well-defined minimum, stabilizing training. - Covariance Matrix:
Positive definite matrices often represent covariance matrices in probabilistic models and Gaussian distributions.
Examples
-
Basic Example:
is symmetric, and for any :
-
Non-positive Definite Example:
Eigenvalues of are , . Since , is not positive definite.