ConceptMSIAI-written · not yet proofread

Least Squares Estimator

Also LS Estimator · Linear Least Squares Estimator · LLS · Least Squares · Method of Least Squares · R_LS

The least squares estimator θ̂_LS minimizes the sum of squared prediction errors Σ (y_k − φ_kᵀθ)² of a model that is linear in the parameters; in the resistance example it is R̂_LS = Σ u_k i_k / Σ i_k².

Know these firstEstimator Invertible Matrix Positive Semidefinite Matrix

Definition

Model the measurements $y_1, \dots, y_N \in \mathbb{R}$ as linear in an unknown parameter $\theta \in \mathbb{R}^d$ ,

y_k = \phi_k^\top\theta + \epsilon_k, \qquad k = 1, \dots, N,

with known regression vectors $\phi_k \in \mathbb{R}^d$ and zero-mean noise $\epsilon_k$ . The least squares estimate $\hat\theta_\text{LS}$ minimizes the sum of squared prediction errors:

\hat\theta_\text{LS} \in \arg\min_{\theta \in \mathbb{R}^d} \sum_{k=1}^N \big(y_k - \phi_k^\top\theta\big)^2 = \arg\min_{\theta \in \mathbb{R}^d} \|y - \Phi\theta\|_2^2,

where $y = [y_1, \dots, y_N]^\top$ and $\Phi \in \mathbb{R}^{N \times d}$ has the rows $\phi_k^\top$ .

Formula#

If $\Phi$ has full column rank ( $\Phi^\top\Phi \succ 0$ , see Rank and Invertibility):

\hat\theta_\text{LS} = (\Phi^\top\Phi)^{-1}\Phi^\top y.

Resistance example ( $y_k = u_k$ , $\phi_k = i_k$ , $\theta = R$ , $d = 1$ ):

\hat R_\text{LS}^{[N]} = \arg\min_{R \in \mathbb{R}} \sum_{k=1}^N (R\,i_k - u_k)^2 = \frac{\frac1N\sum_{k=1}^N u_k\,i_k}{\frac1N\sum_{k=1}^N i_k^2}

(derivation in MSI_VL01 > 1.2.3 Least Squares (LS)).

Properties#

Noise only on the measured outputs: with i.i.d. noise $\epsilon_k$ of variance $\sigma_\epsilon^2$ and exactly known $\phi_k$ , $\hat\theta_\text{LS}$ is unbiased with covariance $\sigma_\epsilon^2(\Phi^\top\Phi)^{-1}$ , and for large $N$ approximately $\hat\theta_\text{LS} \sim \mathcal{N}(\theta_\text{true}, \sigma_\epsilon^2(\Phi^\top\Phi)^{-1})$ (Multivariate Normal Distribution). Details in script chapter 4.
Noise on the regressor ⇒ bias: in the resistance example the regressor $i_k = i_\text{true} + n_{i,k}$ is itself noisy. Then $\lim_{N \to \infty} \hat R_\text{LS}^{[N]} = \frac{u_\text{true}\,i_\text{true}}{i_\text{true}^2 + \sigma_i^2} = \frac{R_\text{true}}{1 + \sigma_i^2/i_\text{true}^2} < R_\text{true}.$ The numerator converges to $u_\text{true} i_\text{true}$ (the cross terms vanish), but the denominator picks up the variance $\sigma_i^2$ of the current noise (Law of Large Numbers). The estimator is not asymptotically unbiased and always underestimates $R$ .
The size of this bias depends only on the current noise relative to the true current, $\sigma_i^2/i_\text{true}^2$ , not on the voltage noise $\sigma_u^2$ . This is a general property of least squares: noise on the regression variable biases the estimate.

Example#

Sources#

MSI script pp. 11–12 (§1.2.3), MSI_VL01 > 1.2.3 Least Squares (LS)
MSI script pp. 20–21 (§2.5.2, bias analysis, eqs. (2.7)–(2.10)), MSI Slides 02 > 2.5.2 Least Squares estimator (not published)
MSI script p. 29 (§4.1, problem formulation), p. 38 (covariance and distribution of $\hat\theta_\text{LS}$ )

Connections

Mentioned in inline links9

Error-in-Variables Estimator
Why it works: both averages are linear in the noise, so the zero-mean noise averages out in each. Only one nonlinear operation, the final division, is applied after averaging. The Simple Approach Estimator does the division first and the L… §
Simple Approach Estimator
So under these optimistic assumptions the estimator overestimates $R$ , in contrast to the Least Squares Estimator, which underestimates it. §
VL01 – Introduction
The concept notes for this lecture now exist. In ## Concepts, replace the plain-text names with links: “Estimator” → Estimator (foundation), “Resistance estimation example” → Resistance Estimation Example, “Error-in-Variables estimator” →… §
Resistance Estimation Example
Estimate the resistance $R$ from $N$ noisy measurements $(u_k, i_k)$ . Three natural estimators, averaging the ratios, dividing the averages and least squares, all converge as $N$ grows, but to three different values (script Fig. 1.4). The… §
Least Squares Estimator — $\sum_k u_k i_k / \sum_k i_k^2$ : minimizes squared voltage residuals, biased downwards §
Asymptotically Unbiased EstimatorPROB
In the resistance example, the Error-in-Variables Estimator is asymptotically unbiased, while the Least Squares Estimator and the Simple Approach Estimator are not. §
Covariance MatrixPROB
In MSI: the least squares estimate with i.i.d. noise of variance $\sigma_\epsilon^2$ has $\operatorname{cov}(\hat\theta_\text{LS}) = \sigma_\epsilon^2(\Phi^\top\Phi)^{-1}$ (Least Squares Estimator). §
EstimatorPROB
From noisy voltage and current measurements $(u_k, i_k)$ , MSI builds three estimators of $R$ : the Simple Approach Estimator $\frac1N\sum_k \frac{u_k}{i_k}$ , the Error-in-Variables Estimator $\frac{\sum_k u_k}{\sum_k i_k}$ and the Least Squ… §
Law of Large NumbersPROB
Take $y_k = n_{i,k}^2$ . Then $\mathbb{E}\{y_k\} = \mathbb{E}\{n_{i,k}^2\} = \sigma_i^2 + 0^2 = \sigma_i^2$ (Variance), so $\frac1N\sum_k n_{i,k}^2 \to \sigma_i^2$ . This term is why the denominator of the Least Squares Estimator converges t… §
Multivariate Normal DistributionPROB
Use in MSI: the least squares estimate with i.i.d. noise of variance $\sigma_\epsilon^2$ is approximately $\hat\theta_\text{LS} \sim \mathcal{N}(\theta_\text{true}, \sigma_\epsilon^2(\Phi^\top\Phi)^{-1})$ for many measurements (Least Squar… §

Search in these mentions →