Derivatives

Rules of Derivatives

Constant Rule

$$\text{[}kf(x){]}^{\prime }=k{f}^{\prime }(x)$$

Proof:

$$\begin{array}{rl}\frac{kf(x+\Delta x)-kf(x)}{\Delta x}& =k\frac{f(x+\Delta x)-f(x)}{\Delta x}\end{array}$$

Sum Rule

\left\[ f(x) + g(x) \right]' = f'(x) + g'(x)

---

Proof

$$\begin{array}{rl}& \frac{\left(f(x+\Delta x)+g(x+\Delta x)\right)-\left(f(x)+g(x)\right)}{\Delta x}\\ & =\frac{f(x+\Delta x)-f(x)}{\Delta x}+\frac{g(x+\Delta x)-g(x)}{\Delta x}\end{array}$$

Product Rule

\left\[ f(x)g(x) \right]' = f'(x)g(x) + f(x)g'(x)

—

Proof

$$\begin{array}{rl}& \frac{f(x+\Delta x)g(x+\Delta x)-f(x)g(x)}{\Delta x}\\ & =\frac{f(x+\Delta x)g(x+\Delta x)-f(x+\Delta x)g(x)+f(x+\Delta x)g(x)-f(x)g(x)}{\Delta x}\\ & =f(x+\Delta x)\frac{g(x+\Delta x)-g(x)}{\Delta x}+g(x)\frac{f(x+\Delta x)-f(x)}{\Delta x}\end{array}$$

Quotient Rule

\left\[ \frac{f(x)}{g(x)} \right]' = \frac{f'(x)g(x) - f(x)g'(x)}{g^{2}(x)}

---

Proof

$$\begin{array}{rl}\frac{\frac{f(x+\Delta x)}{g(x+\Delta x)}-\frac{f(x)}{g(x)}}{\Delta x}& =\frac{f(x+\Delta x)g(x)-f(x)g(x+\Delta x)}{\Delta xg(x+\Delta x)g(x)}\\ & =\frac{f(x+\Delta x)g(x)-f(x)g(x)+f(x)g(x)-f(x)g(x+\Delta x)}{\Delta xg(x+\Delta x)g(x)}\\ & =\frac{g(x)\frac{f(x+\Delta x)-f(x)}{\Delta x}-f(x)\frac{g(x+\Delta x)-g(x)}{\Delta x}}{g(x+\Delta x)g(x)}\end{array}$$

Chain Rule

$$\frac{\partial y}{\partial x}=\frac{\partial y}{\partial u}\frac{\partial u}{\partial x}$$

---

Proof

$$\begin{array}{rl}f(x)-f(a)& =\varphi (x)(x-a)\\ g(x)-g(a)& =\psi (x)(x-a)\\ & \\ f(g(x))-f(g(a))& =\varphi (g(x))\cdot \text{[}g(x)-g(a)]\\ & =\varphi (g(x))\cdot \psi (x)(x-a)\end{array}$$

By applying Composition Rule:

$$\begin{array}{rl}\lim \_x\to a\frac{f(g(x))-f(g(a))}{x-a}& =\lim \_x\to a\varphi (g(x))\cdot \psi (x)\\ & =\lim \_x\to a\varphi \left(g(x)\right)\cdot \lim \_x\to a\psi (x)\\ & =\lim \_u\to g(a)\varphi (u)\cdot \lim \_x\to a\psi (x)\\ & =\lim \_u\to g(a)\frac{f(u)-f(g(a))}{u-g(a)}\lim \_x\to a\frac{g(x)-g(a)}{x-a}\\ & =f^{\prime }(g(a))g^{\prime }(a)\end{array}$$

Differential

if $y$ is determined by $x$, ($y=f(x)$) then the differential of y ($dy$) is $f^{\prime }(x),dx$. if $x$ is a free variable, then $dx$ is an independent variable, representing a change of $x$.

$$\Delta y=A(x)\Delta x+o(\Delta x)$$

the little o here represent an infinitesimal which is strictly higher order of $\Delta x$. little o

if $f$ is differentiable then it’s derivable

$$f^{\prime }(x):=\lim \_\Delta x\to 0\frac{\Delta y}{\Delta x}=A(x)$$

if $f$ is derivable then it’s differentiable:

$$\frac{\Delta y}{\Delta x}=f^{\prime }(x)+o(1)⟹\Delta y=f^{\prime }(x)\Delta x+o(\Delta x)$$

thus, derivable is equivalent to differentiable.

Total Derivative

Total Differential

Total differentiable，if：

$$\Delta z=\nabla f\cdot \Delta P+o(\rho )$$

if all partial derivatives are continuous, then total differentiable

$$\begin{array}{rl}\Delta z& =f(P+\Delta P)-f(P)\\ & =f\_x(P)\Delta x+f\_y(P)\Delta y+(ϵ\_1\Delta x+ϵ\_2\Delta y)\\ & =f\_x(P)\Delta x+f\_y(P)\Delta y+o(\rho )\\ & =\nabla f\cdot \Delta P+o(\rho )\end{array}$$

Directional Derivative

rate of change along this line/direction $u$:

$$D\_uf=\lim \_h\to 0\frac{\Delta z}{h}=\lim \_h\to 0\frac{f(P+hu)-f(P)}{h}=\nabla f\cdot u$$

Gradient

gradient is a partial derivative vector

$$\left[\begin{array}{c}\frac{\partial y}{\partial x\_1}\\ \frac{\partial y}{\partial x\_2}\\ ⋮\\ \frac{\partial y}{\partial x\_n}\end{array}\right]$$

Hessian Matrix

Hessian Matrix

Elements of Hessian Matrix is the second derivative of $i$-th then $j$-th input.

$$H\_i,j=\frac{{\partial }^{2}f}{\partial x\_ix\_j}$$

Jacobian Matrix

Jacobian matrix

Jacobian matrix is the partial derivative matrix for a multi-input-output function. The $i,j$ elementh is the the derivative of $i$-th output w.r.t $j$-th input

$$J\_i,j=\frac{\partial f\_i}{\partial x\_j}$$