Point Estimation

Concepts#

1. 1 Estimatora function of the data; the estimate is random, the parameter is not start here
2. 2 Unbiased Estimatorright on average for every $N$ (Definition 1)
3. 3 Asymptotically Unbiased Estimatorright on average as $N \to \infty$ (Definition 2)
4. 4 Consistent Estimatorthe estimate itself concentrates at the true value (Definition 3)
5. 5 Sample Meanthe unbiased, consistent estimator of a mean, variance $\sigma^2/N$
6. 6 Sample Variancethe unbiased estimator of a variance, dividing by $N - 1$
7. 7 Law of Large Numbersaverages converge to expectations; the tool for asymptotic analysis

How it fits together#

Bias and spread are separate questions. Bias concerns the mean of $\hat\theta(y)$, performance concerns its covariance (Moments of Random Variables). An estimator can be unbiased but scatter widely, or be slightly biased with small spread. Among unbiased estimators, $\operatorname{cov}(\hat\theta^A) \succcurlyeq \operatorname{cov}(\hat\theta^B)$ makes $\hat\theta^B$ the better one.

The three definitions form a chain:

Unbiasedness and asymptotic unbiasedness are statements about averages over many repeated experiments. Consistency is a statement about the single estimate you actually compute: with enough data, it is close to $\theta_\text{true}$ with high probability. That is why the script calls it “stronger and even more desirable”.

The worked examples, side by side (independent measurements with mean $\mu_y$ and variance $\sigma_y^2$):

The sample mean additionally has variance $\sigma_y^2/N \to 0$, so it is consistent. The middle row shows where bias comes from: the deviations are measured from $M(y)$, which depends on the same data, so the cross term $\mathbb{E}\{y_k M(y)\}$ contains $\sigma_y^2/N$ and not just $\mu_y^2$.

Where the computations come from. Expectations of linear estimators follow from linearity alone. Anything with squares or products needs $\mathbb{E}\{y^2\} = \sigma^2 + \mu^2$ and the factorization $\mathbb{E}\{y_k y_j\} = \mu^2$ for independent $j \ne k$. Limits of estimators built from averages are evaluated with the Law of Large Numbers, replacing each average by its expectation.

Plausibility is not enough. The MSI Resistance Estimation Example has three reasonable estimators that converge to three different values. Only the analysis above, under an explicit noise model, shows which one is asymptotically unbiased.

Key Formulas#

• $\mathbb{E}\{\hat\theta(y)\} = \theta_\text{true}$ (Unbiased Estimator), $\lim_{N\to\infty}\mathbb{E}\{\hat\theta(y_1, \dots, y_N)\} = \theta_\text{true}$ (Asymptotically Unbiased Estimator)
• $\operatorname{Var}(M(y)) = \sigma_y^2/N$ (Sample Mean)
• $S^2 = \frac{1}{N-1}\sum_k(y_k - M(y))^2$ (Sample Variance)

Lectures#

• No write-up yet. MSI covers this material in script §2.4 (MSI Slides 02 > 2.4 Statistical Estimators (not published)), planned for the lectures of 2026-10-28 and 2026-11-04. MSI_VL01 introduces the notion of an estimator with the resistance example.

Related Topics#

• Moments of Random Variables — bias and performance are the first two moments of the estimate
• Resistance Estimation Example — the MSI case study these definitions are made for