Definition
A square Matrix (or ) is positive definite if:

  1. is symmetric (or Hermitian if is complex): (or ).
  2. For all non-zero vectors (or ):

    (In the complex case, , where is the conjugate transpose of .)

Intuition

  • A positive definite matrix corresponds to a quadratic form that always produces positive values for non-zero .
  • It represents a multidimensional generalization of a convex parabola, where the energy (or quadratic form) reaches a unique minimum, often associated with optimization and stability.

In Deep Learning, a positive definite Hessian matrix (second-derivative matrix) of a loss function guarantees that the function is convex, ensuring the existence of a unique minimum.

Key Properties

  1. Eigenvalues:
    All eigenvalues of are positive: .
  2. Determinant and Leading Principal Minors:
    • .
    • All leading principal minors (determinants of top-left submatrices) are positive.
  3. Cholesky Decomposition:
    A positive definite matrix can be decomposed as:

    where is a lower triangular matrix with positive diagonal entries.
  4. Matrix Inversion:
    A positive definite matrix is always invertible.
  5. Norm Property:
  6. Relation to Convexity:
    • If is the Hessian of a scalar function , i.e., , then is strictly convex.

Applications in Deep Learning

  1. Optimization:
    • Positive definite matrices arise in the Hessian of loss functions. If the Hessian is positive definite, the loss is convex, and gradient descent converges to the unique minimum.
    • Convexity ensures no local minima exist, making optimization simpler and more predictable.
  2. Energy Landscape:
    In machine learning models, ensuring a positive definite structure means the “energy” (loss function) has a well-defined minimum, stabilizing training.
  3. Covariance Matrix:
    Positive definite matrices often represent covariance matrices in probabilistic models and Gaussian distributions.

Examples

  1. Basic Example:

    is symmetric, and for any :

  2. Non-positive Definite Example:

    Eigenvalues of are , . Since , is not positive definite.