Common Vector and Matrix Derivatives Cheat Sheet

• categories: Linear algebra, Cheat Sheet

1. Scalar with Respect to a Vector

1. Gradient of a Linear Function
   $f(\mathbf{x})={\mathbf{a}}^T\mathbf{x}$

   $${\nabla }_{\mathbf{x}}f=\mathbf{a}.$$

2. Gradient of a Quadratic Function
   $f(\mathbf{x})=\frac{1}{2}{\mathbf{x}}^T\mathbf{Ax}$ (where $\mathbf{A}$ is symmetric)

   $${\nabla }_{\mathbf{x}}f=\mathbf{Ax}.$$

3. Gradient of a Norm
   $f(\mathbf{x})=\Vert \mathbf{x}{\Vert }_2=\sqrt{{\mathbf{x}}^T\mathbf{x}}$

   $${\nabla }_{\mathbf{x}}f=\frac{\mathbf{x}}{\Vert \mathbf{x}{\Vert }_2}.$$

4. Gradient of the Dot Product
   $f(\mathbf{x})={\mathbf{a}}^T\mathbf{x}$

   $${\nabla }_{\mathbf{x}}f=\mathbf{a}.$$

5. Gradient of the Log-Sum-Exp Function
   $f(\mathbf{x})=\log \left({\sum }_i{e}^{x_i}\right)$

   $${\nabla }_{\mathbf{x}}f=\frac{\exp (\mathbf{x})}{{\sum }_i\exp (x_i)}.$$

---

2. Vector with Respect to a Vector

1. Derivative of a Vector-Vector Product
   $\mathbf{f}(\mathbf{x})=\mathbf{Ax}$

   $$\frac{\partial \mathbf{f}}{\partial \mathbf{x}}=\mathbf{A}.$$

2. Derivative of a Norm-Squared
   $f(\mathbf{x})=\Vert \mathbf{x}{\Vert }_2^2={\mathbf{x}}^T\mathbf{x}$

   $${\nabla }_{\mathbf{x}}f=2\mathbf{x}.$$

---

3. Scalar with Respect to a Matrix

1. Derivative of Trace
   $f(\mathbf{A})=\text{tr}(\mathbf{A})$

   $$\frac{\partial f}{\partial \mathbf{A}}=\mathbf{I}.$$

2. Derivative of a Quadratic Form
   $f(\mathbf{A})={\mathbf{x}}^T\mathbf{Ax}$

   $$\frac{\partial f}{\partial \mathbf{A}}=\mathbf{x}{\mathbf{x}}^T.$$

3. Derivative of Trace with Matrix Multiplication
   $f(\mathbf{A})=\text{tr}(\mathbf{AB})$

   $$\frac{\partial f}{\partial \mathbf{A}}={\mathbf{B}}^T.$$

---

4. Matrix with Respect to a Matrix

1. Derivative of a Matrix Product
   $\mathbf{C}=\mathbf{AB}$

   $$\frac{\partial \mathbf{C}}{\partial \mathbf{A}}={\mathbf{B}}^T.$$

2. Derivative of a Quadratic Form
   $f(\mathbf{A})=\text{tr}({\mathbf{A}}^T\mathbf{BA})$

   $$\frac{\partial f}{\partial \mathbf{A}}=\mathbf{BA}+{\mathbf{B}}^T\mathbf{A}.$$

---

5. Special Functions

1. Softmax Function
   $\sigma (\mathbf{x}{)}_i=\frac{e^{x_i}}{{\sum }_j{e}^{x_j}}$

   $$\frac{\partial \sigma (\mathbf{x}{)}_i}{\partial x_j}=\sigma (\mathbf{x}{)}_i({\delta }_{ij}-\sigma (\mathbf{x}{)}_j).$$

2. Sigmoid Function
   $s(x)=\frac{1}{1+e^{-x}}$

   $$\frac{\partial s(x)}{\partial x}=s(x)(1-s(x)).$$