ConceptProbability and Statistics used inMSIAI-written · not yet proofread

Covariance Matrix

Also Covariance Operator · cov · Variance-Covariance Matrix · Kovarianzmatrix

The covariance matrix Σ_X = cov(X) = E{(X − μ_X)(X − μ_X)ᵀ} of a random vector X ∈ ℝⁿ collects all variances (diagonal) and covariances (off-diagonal) of its components in one symmetric positive semidefinite n × n matrix.

Know these firstCovariance Expected Value Positive Semidefinite Matrix

Formula#

\operatorname{cov}(X) = \mathbb{E}\{XX^\top\} - \mu_X\mu_X^\top, \qquad \operatorname{cov}(QX) = Q\,\operatorname{cov}(X)\,Q^\top.

ProofLinear transformation

Y = QX

Q \in \mathbb{R}^{m \times n}

(MSI script)

$\mu_Y = \mathbb{E}\{QX\} = Q\mu_X$ , so

\begin{aligned} \Sigma_Y &= \mathbb{E}\{(QX - Q\mu_X)(QX - Q\mu_X)^\top\} = \mathbb{E}\{Q(X - \mu_X)(X - \mu_X)^\top Q^\top\} \\ &= Q\,\mathbb{E}\{(X - \mu_X)(X - \mu_X)^\top\}\,Q^\top = Q\Sigma_X Q^\top. \end{aligned}

If $m = n$ and $Q$ is orthogonal ( $Q^{-1} = Q^\top$ ): $\Sigma_Y^{-1} = (Q^\top)^{-1}\Sigma_X^{-1}Q^{-1} = Q\Sigma_X^{-1}Q^\top$ .

Properties#

Symmetric and positive semidefinite: $\Sigma_X = \Sigma_X^\top$ and $z^\top\Sigma_X z = \mathbb{E}\{(z^\top(X - \mu_X))^2\} = \operatorname{Var}(z^\top X) \ge 0$ for all $z$ (Positive Semidefinite Matrix). The Multivariate Normal Distribution requires $\Sigma_X \succ 0$ .
Independent components give a diagonal covariance matrix (all off-diagonal covariances vanish).
Covariance of an estimator: for an unbiased estimator $\hat\theta(y)$ , $\operatorname{cov}(\hat\theta(y))$ measures its performance; smaller is better. Two unbiased estimators are compared in the matrix sense: $\operatorname{cov}(\hat\theta^A) \succcurlyeq \operatorname{cov}(\hat\theta^B)$ means $\hat\theta^B$ is better.
In MSI: the least squares estimate with i.i.d. noise of variance $\sigma_\epsilon^2$ has $\operatorname{cov}(\hat\theta_\text{LS}) = \sigma_\epsilon^2(\Phi^\top\Phi)^{-1}$ (Least Squares Estimator).

Example#

Sources#

MSI script pp. 16–17 (§2.3.1–2.3.2), MSI Slides 02 > 2.3.1 Mean and Covariance Matrix (not published), MSI Slides 02 > 2.3.2 Multidimensional Normal Distribution (not published)
MSI script p. 18 (comparing estimators by their covariance), MSI Slides 02 > 2.4 Statistical Estimators (not published)
MSI script p. 38 (covariance of the LS estimate)

Connections

Mentioned in inline links9

VL02 – Probability and Statistics in a NutshellMSI
The mean $\mu_X \in \mathbb{R}^n$ is a vector. The variance generalises to the Covariance Matrix $\Sigma_X = \mathrm{cov}(X) = \mathbb{E}\{(X-\mu_X)(X-\mu_X)^\top\}$ , which holds all variances and covariances. The vector analogue of §2.2.1… §
Covariance Matrix: all variances and covariances of a random vector §
Consistent Estimator
its covariance tends to zero as $N \to \infty$ . §
Covariance
Vector form: all variances and covariances of a random vector are collected in the Covariance Matrix, with entries $(\Sigma_X)_{jk} = \sigma(X_j, X_k)$ . §
Estimator
small spread: the covariance $\operatorname{cov}(\hat\theta(y))$ . Of two unbiased estimators $\hat\theta^A$ , $\hat\theta^B$ , $\hat\theta^B$ performs better if $\operatorname{cov}(\hat\theta^A) \succcurlyeq \operatorname{cov}(\hat\theta^B)$ ; §
Expected Value
Higher moments: Variance, Covariance and the Covariance Matrix are all expectations of products of centred variables $X - \mu_X$ . §
Multivariate Normal Distribution
Moments: $\mathbb{E}\{X\} = \mu$ and $\operatorname{cov}(X) = \Sigma$ (by integral computations), so the mean and the covariance matrix determine the distribution. §
(derivation in Covariance Matrix). If $m = n$ and $Q$ is orthogonal, $\Sigma_Y^{-1} = Q\Sigma_X^{-1}Q^\top$ . §
Unbiased Estimator
Unbiasedness must hold for the given number $N$ of measurements, and for every value $\theta_\text{true}$ could take. It says nothing about the spread: an unbiased estimator can still have a large covariance. §
Variance
Generalizations: the variance is the Covariance of a variable with itself, $\sigma(X, X) = \sigma_X^2$ , and the diagonal of the Covariance Matrix. §
Moments of Random Variables
Instead of a whole density, one usually works with a few numbers: the mean (where the variable lies), the variance (how far it spreads), the covariance and correlation (how two variables move together), and for random vectors the covarianc… §
Covariance Matrix — all variances and covariances of a random vector §

Search in these mentions →