ConceptProbability and Statistics used inMSIAI-written · not yet proofread

Law of Large Numbers

Also LLN · Weak Law of Large Numbers · Strong Law of Large Numbers · Gesetz der großen Zahlen

The law of large numbers states that the average of an infinite sequence of measurements y_k of a random variable y converges to its expected value, lim_{N→∞} (1/N) Σ y_k = E{y}.

Know these firstExpected Value Sample Mean

Properties#

Applies to functions of the data: applying it to $f(y_k)$ $f (y_{k})$ gives $\frac1N\sum_k f(y_k) \to \mathbb{E}\{f(y)\}$ $\frac{1}{N} \sum_{k} f (y_{k}) \to E {f (y)}$ . With the MSI noise model $u_k = u_\text{true} + n_{u,k}$ $u_{k} = u_{true} + n_{u, k}$ , $i_k = i_\text{true} + n_{i,k}$ $i_{k} = i_{true} + n_{i, k}$ (independent, zero-mean, variances $\sigma_u^2, \sigma_i^2$ $σ_{u}^{2}, σ_{i}^{2}$ ):
- $\frac1N\sum_k u_k \to u_\text{true}$ and $\frac1N\sum_k n_{i,k} \to 0$ ,
- $\frac1N\sum_k n_{i,k} n_{u,k} \to 0$ (independent and zero-mean, so the product has zero expectation),
- $\frac1N\sum_k n_{i,k}^2 \to \sigma_i^2$ (the mean of the squares is the variance).
Main tool for asymptotic analysis: limits of estimators built from averages are computed by replacing each average by its expected value. Ratios of averages converge to ratios of expectations (if the denominator’s limit is nonzero), which is how the Error-in-Variables Estimator is shown to converge to $R_\text{true}$ .
Relation to consistency: it says that the Sample Mean is a consistent estimator of the mean.

Example#

Sources#

MSI script pp. 19–20 (§2.5), MSI Slides 02 > 2.5 Analysis of the Resistance Estimation Example (not published)
MSI script p. 38 (asymptotic normality of LS estimates)

Connections

Mentioned in inline links12

Error-in-Variables EstimatorMSI
Asymptotically unbiased under the noise model $u_k = u_\text{true} + n_{u,k}$ , $i_k = i_\text{true} + n_{i,k}$ with zero-mean, independent, i.i.d. noise. By the Law of Large Numbers, numerator and denominator converge separately: §
Least Squares EstimatorMSI
The numerator converges to $u_\text{true} i_\text{true}$ (the cross terms vanish), but the denominator picks up the variance $\sigma_i^2$ of the current noise (Law of Large Numbers). The estimator is not asymptotically unbiased and always… §
Simple Approach EstimatorMSI
Multiplying by $u_k = u_\text{true} + n_{u,k}$ and averaging, the terms linear in the noise and the product $n_{i,k} n_{u,k}$ vanish, but $n_{i,k}^2$ averages to $\sigma_i^2$ (Law of Large Numbers): §
Resistance Estimation ExampleMSI
Estimate the resistance $R$ from $N$ noisy measurements $(u_k, i_k)$ . Three natural estimators, averaging the ratios, dividing the averages and least squares, all converge as $N$ grows, but to three different values (script Fig. 1.4). The… §
The noise model (script §2.5): $u_k = u_\text{true} + n_{u,k}$ , $i_k = i_\text{true} + n_{i,k}$ , with $n_{u,k}$ , $n_{i,k}$ independent of each other, i.i.d., zero mean, variances $\sigma_u^2$ , $\sigma_i^2$ , and… §
Asymptotically Unbiased Estimator
Easiest property to analyse: for estimators built from averages, the limits can be evaluated with the Law of Large Numbers. This is how MSI decides between the three resistance estimators. §
Consistent Estimator
The Sample Mean of $N$ independent measurements is unbiased (so asymptotically unbiased) and has variance $\sigma_y^2/N \to 0$ . By the sufficient condition it is consistent: with enough data the average is arbitrarily close to $\mu_y$ with… §
Expected Value
Averages approach the expected value: $\frac1N\sum_{k=1}^N y_k \to \mathbb{E}\{y\}$ for many measurements (Law of Large Numbers). §
Normal Distribution
The approximate normality of averages of many independent errors is the central limit theorem. The script attributes it to “a law of large numbers” (p. 38); the Law of Large Numbers proper only says that averages converge to the mean. §
Sample Mean
Consistent: unbiased with variance $\to 0$ (Consistent Estimator). For $N \to \infty$ , $M(y) \to \mu_y$ (Law of Large Numbers). §
Statistical Independence
In the MSI resistance example the noises $n_{u,k}$ on the voltage and $n_{i,k}$ on the current are independent and zero-mean. Hence $\mathbb{E}\{n_{i,k} n_{u,k}\} = \mathbb{E}\{n_{i,k}\}\mathbb{E}\{n_{u,k}\} = 0$ , and the average of the pr… §
Variance
Noise of a measurement: in MSI, the noise on the current and the voltage has variances $\sigma_i^2$ and $\sigma_u^2$ , and $\frac1N\sum_k n_{i,k}^2 \to \sigma_i^2$ (Law of Large Numbers). §
Point Estimation
An estimator turns random measurements into an estimate of a fixed but unknown parameter, so the estimate is a random variable. Two questions judge it: is it right on average (unbiased, or at least asymptotically unbiased), and how much do… §
Law of Large Numbers — averages converge to expectations; the tool for asymptotic analysis §

Search in these mentions →