Variance

Variance of Single RV

$$\mathrm{Var}\text{[}X]:=\mathbb{E}\text{[}(X-\mu {)}^2]$$

applying Linearity and Distributivity w.r.t addition:

$$\begin{array}{rl}\mathrm{Var}\text{[}X]& =\mathbb{E}\text{[}{X}^2-2\mu X+{\mu }^2]\\ & =\mathbb{E}\text{[}{X}^2]-2\mu \mathbb{E}\text{[}X]+{\mu }^2\\ & =\mathbb{E}\text{[}{X}^2]-{\mu }^2\end{array}$$

Transformation

$Y=CX$

$$\sigma \_{CX}^2=C^2\sigma \_X^2$$

Proof:

$$\begin{array}{rl}{\sigma }_Y^2& =E\text{[}{Y}^2]-E^2\text{[}Y]\\ & =E\text{[}{C}^2{X}^2]-E\text{[}CX{]}^2\\ & =C^2(E\text{[}{X}^2]-E\text{[}X{]}^2)\\ & =C^2{\sigma }_X^2\end{array}$$

$Y=X+C$

Suppose $Y=X+C$, $C$ is a constant:

$$\sigma \_{X+C}^2=\sigma \_X^2$$

Proof 1:

Given (Proofs are in Linearity and Transformation):

$$\begin{array}{rl}\mu \_Y& =\mu \_X+C\\ f\_Y(y)& =f\_X(y-C)\end{array}$$

Thus:

\begin{align} \sigma\_{Y}^{2} & = E\_{Y}\[(Y-\mu\_{Y})^{2}] \\ & = \int\_{-\infty}^{\infty} (y-\mu\_{Y})^{2} f\_{Y}(y) , dy \\ & = \int\_{-\infty}^{\infty} (y - \mu\_{Y})^{2} f\_{X}(y-C) , dy \\ & = \int\_{-\infty}^{\infty} (s+C - (\mu\_{X}+C))^{2} f\_{X}(s) , ds \\ & = E\_{X}\left\[(X-\mu\_{X})^{2}\right] = \sigma\_{X}^{2} \end{align}

Proof2:

with LOTUS the proof is simpler:

\begin{align} \sigma\_{Y}^{2} & = E\[(Y-\mu\_{Y})^{2}] \\ & = E\left{ \left\[ \left( X+C \right) - \left( \mu\_{X}+C \right) \right]^{2} \right} \\ & = E\left\[ \left( X-\mu\_{X} \right)^{2} \right] = \sigma\_{X}^{2} \end{align}

Sum Rule

\begin{align} \mathrm{Var\[X+Y]} &= E\left{ \left\[ (X+Y) - (\mu\_{X}+\mu\_{Y})\right]^{2} \right} \\ &= \sigma\_{X}^{2}+\sigma\_{Y}^{2}+2\mathrm{Cov}(X,Y) \end{align}

Covariance

variance between two r.v.

$$\mathrm{Cov}(X,Y)=\mathbb{E}\text{[}(X-\mu \_X)(Y-\mu \_Y)]$$

also:

$$\begin{array}{rl}\mathrm{Cov}(X,Y)& =\mathbb{E}\text{[}XY-\mu \_XY-\mu \_YX+\mu \_X\mu \_Y]\\ & =\mathbb{E}\text{[}XY]-\mu \_X\mu \_y\end{array}$$

Covariance Matrix