Hessian matrix

• categories: Real Analysis, Definition

Definition:
The Hessian matrix is a square matrix of second-order partial derivatives of a scalar-valued function. It describes the local curvature of the function and plays a crucial role in optimization and machine learning.

For a twice-differentiable function $f:{\mathbb{R}}^n\to \mathbb{R}$, the Hessian matrix $H_f(x)$ is defined as:

$$H_f(x)=\left[\begin{array}{cccc}\frac{{\partial }^{2}f}{\partial x_1^2}& \frac{{\partial }^{2}f}{\partial x_1\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_1\partial x_n}\\ \frac{{\partial }^{2}f}{\partial x_2\partial x_1}& \frac{{\partial }^{2}f}{\partial x_2^2}& \cdots & \frac{{\partial }^{2}f}{\partial x_2\partial x_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial x_n\partial x_1}& \frac{{\partial }^{2}f}{\partial x_n\partial x_2}& \cdots & \frac{{\partial }^{2}f}{\partial x_n^2}\end{array}\right].$$

• The $(i,j)$ entry of $H_f(x)$ is $\frac{{\partial }^{2}f}{\partial x_i\partial x_j}$.
• If $f$ is twice continuously differentiable, $H_f(x)$ is symmetric.

---

Key Properties

1. Symmetry:
   If $f$ is twice continuously differentiable, the mixed partial derivatives are equal ($\frac{{\partial }^{2}f}{\partial x_i\partial x_j}=\frac{{\partial }^{2}f}{\partial x_j\partial x_i}$), so $H_f(x)$ is symmetric.

2. Curvature:
   The Hessian describes the curvature of $f$ at a point $x$:

   • Positive eigenvalues indicate convexity in those directions.
   • Negative eigenvalues indicate concavity.

3. Quadratic Approximation:
   Near a point $x_0$, $f(x)$ can be approximated using its Taylor expansion:
   $f(x)\approx f(x_0)+\nabla f(x_0{)}^{\top }(x-x_0)+\frac{1}{2}(x-x_0{)}^{\top }{H}_f(x_0)(x-x_0).$
   Here, $H_f(x_0)$ determines the curvature of the approximation.

---

Uses of the Hessian Matrix

1. Optimization:

   • Gradient Descent: The Hessian is used in second-order methods like Newton’s method to adjust the step size and direction:
     $x\leftarrow x-H_f(x{)}^{-1}\nabla f(x).$
   • Convexity Analysis:
     • If $H_f(x)$ is positive definite at $x$, $f$ has a local minimum.
     • If $H_f(x)$ is negative definite, $f$ has a local maximum.
     • If $H_f(x)$ has mixed eigenvalues, $x$ is a saddle point.

2. Data Science

Eigenvalues and Critical Points

1. Positive Definite:

   • All eigenvalues of $H_f(x)$ are positive.
   • $f$ has a local minimum at $x$.

2. Negative Definite:

   • All eigenvalues of $H_f(x)$ are negative.
   • $f$ has a local maximum at $x$.

3. Indefinite:

   • $H_f(x)$ has both positive and negative eigenvalues.
   • $x$ is a saddle point.

4. Positive Semidefinite / Negative Semidefinite:

   • All eigenvalues are $\ge 0$ or $\le 0$, respectively.
   • $f$ may have a minimum/maximum, but the point may not be strict.

---

Numerical Stability and Approximations

1. Hessian-Free Methods:
   Computing $H_f(x)$ directly can be computationally expensive for large $n$. Approximations like finite differences or low-rank representations are used in practice.

2. Quasi-Newton Methods:
   Algorithms like BFGS approximate the Hessian iteratively to avoid computing it explicitly.