Moments of Random Variables

Concepts#

1. 1 Expected Valuethe linear operator $\mathbb{E}\{\cdot\}$ behind all moments start here
2. 2 Variancethe spread of one variable
3. 3 Covariancethe joint spread of two variables
4. 4 Correlationcovariance normalized to $[-1, 1]$
5. 5 Covariance Matrixall variances and covariances of a random vector
6. 6 Multivariate Normal Distributionthe distribution that is fully described by mean and covariance matrix

How it fits together#

One operator, three levels. All moments are expectations of products of centred variables:

Each row contains the one above it: $\sigma(X, X) = \sigma_X^2$, and $\Sigma_X$ has the variances on its diagonal and the covariances off it. All three shortcut formulas come from the same step: expand the product and pull the constant means out of $\mathbb{E}$ by linearity.

Linear maps transform the moments predictably, nonlinear maps don’t.

• $\mathbb{E}\{a + bX\} = a + b\mu_X$, $\operatorname{Var}(a + bX) = b^2\sigma_X^2$, and in vector form $\mathbb{E}\{QX\} = Q\mu_X$, $\operatorname{cov}(QX) = Q\Sigma_X Q^\top$.
• For nonlinear $f$, $\mathbb{E}\{f(X)\} \ne f(\mathbb{E}\{X\})$. The gap is the variance: $\mathbb{E}\{X^2\} = \mu_X^2 + \sigma_X^2$. This one identity explains why the least squares and simple approach estimators in MSI are biased (a $\sigma_i^2$ appears where the noise enters squared), see Resistance Estimation Example.

Independence ⇒ uncorrelated ⇒ products factorize. Independent variables have $\mathbb{E}\{YZ\} = \mu_Y\mu_Z$, hence zero covariance and correlation. This is the workhorse in computations with many measurements: in $\mathbb{E}\{(\sum_k y_k)^2\}$, the $N$ squares contribute $\sigma^2 + \mu^2$ each and the $N(N-1)$ cross terms only $\mu^2$, which gives the $\sigma^2/N$ variance of the Sample Mean. The converse fails: zero correlation only rules out linear dependence (except for jointly Gaussian variables).

Units and normalization. The covariance carries the product of the units, so its size is not comparable across pairs of variables. The Correlation divides by both standard deviations and lands in $[-1, 1]$, with $\pm 1$ for exact linear relations.

Why matrices. For a parameter vector $\theta \in \mathbb{R}^d$ the uncertainty of an estimate is not one number: errors in different components can be correlated. The covariance matrix is symmetric positive semidefinite, and “smaller uncertainty” becomes the matrix inequality $\operatorname{cov}(\hat\theta^A) \succcurlyeq \operatorname{cov}(\hat\theta^B)$ (Point Estimation).

Key Formulas#

• $\sigma_X^2 = \mathbb{E}\{X^2\} - \mu_X^2$ (Variance)
• $\rho(Y, Z) = \frac{\sigma(Y, Z)}{\sigma_Y\sigma_Z} \in [-1, 1]$ (Correlation)
• $\operatorname{cov}(QX) = Q\Sigma_X Q^\top$ (Covariance Matrix)

Lectures#

• No write-up yet. MSI covers this material in script §2.2.1, §2.2.4 and §2.3 (MSI Slides 02 (not published)), planned for the lectures of 2026-10-21 and 2026-10-28.

Related Topics#

• Random Variables and Distributions — the densities these moments summarize
• Point Estimation — bias is a statement about the mean of an estimate, performance about its covariance