Expectation

Definition of Expectation

For discrete random variable

$$\mathbb{E}[X]:=\sum _{x\in X}xP(X=x)$$

For continuous random variable

$$\mathbb{E}[X]:={\int }_{-\infty }^{\infty }xf(x)dx$$

LOTUS

LOTUS (Law of the Unconscious Statistician) Theorem:

$$\begin{array}{rlrl}& \mathbb{E}[g(X)]=\sum _{x}g(x)P(X=x)& \text{if descrete}& \\ & \mathbb{E}[g(X)]={\int }_{-\infty }^{\infty }g(x)f_X(x)dx& \text{if continuous}& \end{array}$$

Lemma: Transformation of probability density

It’s all about sign cancelling

the probability density function is defined as:

$$p_Y(y)=\frac{d}{dy}{F}_Y(y)=\frac{d}{dy}P(Y\le y)$$

given $y=g(x)$, to transform $P(Y\le y)$ to probability about $X$, slice $g$.

if $g$ is monotonically decrease in $[y_1,y_2]$

$$P(y_1<Y\le y_2)=P(g^{-1}(y_2)<X\le g^{-1}(y_1))=P(X\le g^{-1}(y_1))-P(X\le g^{-1}(y_2))$$

if $g$ is monotonically increase in $[y_1,y_2]$:

$$P(y_1<Y\le y_2)=P(X\le g^{-1}(y_2))-P(X\le g^{-1}(y_1))$$

take derivative of $x$, apply chain rule:

$$\frac{d}{dy}{\int }_{-\infty }^x{p}_x(t)dt=p_x(x)\frac{dx}{dy}$$

and both cases generate the same expression:

$$\frac{d}{dx}P(y_1<Y\le y_2)=\frac{p_x(x_2)}{\left|g^{\prime }(y_2)\right|}-\frac{p_x(x_1)}{\left|g^{\prime }(y_1)\right|}$$

So for piecewise monotonic $g$:

$$p_Y(y)=\frac{d}{dy}\sum _{k}P(a_k<Y\le b_k)=p_X(x)\left|\frac{dx}{dy}\right|$$

Proof of LOTUS (continuous)

this leads to LOTUS for continuous variable:

the potential negative sign of $\frac{dy}{dx}$ also cancel with the potential flipping of integral limit caused by substituting variables in definite integral：

$$\mathbb{E}[y]={\int }_{-\infty }^{\infty }y{p}_Y(y)dy={\int }_{-\infty }^{\infty }g(x)p_X(x)\left|\frac{dx}{dy}\right|d\left(\left|\frac{dy}{dx}\right|x\right)={\int }_{-\infty }^{\infty }g(x)p_X(x)dx$$

For discrete variable it’s pretty intuitive:

Since:

$$P(Y=y)=\sum _{x\in X:g(x)=y}P(X=x)$$

So:

$$\mathbb{E}[Y]=\sum _{y\in Y}yP(Y=y)=\sum _{x\in X}g(x)P(X=x)$$

Properties of Expectation

Linearity

Given by LOTUS and linearity of Riemann Integral

applying LOTUS:

$$\mathbb{E}[aX]={\int }_{X}axf(x)dx=a{\int }_{X}xf(x)dx=a\mathbb{E}[X]$$ $$\mathbb{E}[aX]=\sum _{x\in X}axf(x)dx=a\sum _{x\in X}xf(x)dx=a\mathbb{E}[X]$$ $$\mathbb{E}[X+b]={\int }_X(x+b)f(x)dx={\int }_{X}xf(x)dx+b{\int }_{X}f(x)dx=\mathbb{E}[x]+b$$ $$\mathbb{E}[X+b]=\sum _{x\in X}(x+b)f(x)=\sum _{x\in X}xf(x)+b\sum _{x\in X}f(x)=\mathbb{E}[X]+b$$

In conclusion:

$$\mathbb{E}[aX+b]=a\mathbb{E}[X]+b$$

Sum Rule

Distributivity w.r.t addition

Expectation is also distributable w.r.t addition:

$$\begin{array}{rl}\mathbb{E}[X+Y]& ={\int }_Y{\int }_X(x+y)f(x,y)dxdy\\ & ={\int }_Y\left({\int }_{X}x(fx,y)dx+y{\int }_{X}f(x,y)dx\right)dy\\ & ={\int }_Y{\int }_{X}xf(x,y)dxdy+{\int }_{Y}y{f}_Y(y)dy\\ & ={\int }_X\left(x{\int }_Y(fx,y)dy\right)dx+\mathbb{E}[Y]\\ & =\mathbb{E}[X]+\mathbb{E}[Y]\end{array}$$

Product Rule

Independency required

Distributivity w.r.t. multiplication while independent

Given $f(x,y)=f_X(x)f_Y(y)$, Expectation is distributable w.r.t multiplication:

$$\begin{array}{rl}\mathbb{E}[XY]& ={\int }_X{\int }_{Y}xyf(x,y)dxdy\\ & ={\int }_{X}x{f}_X(x){\int }_{Y}y{f}_Y(y)dydx\\ & =\mathbb{E}[Y]{\int }_{X}x{f}_X(x)dy\\ & =\mathbb{E}[X]\mathbb{E}[Y]\end{array}$$