ConceptProbability and Statistics used inMSIAI-written · not yet proofread

Multivariate Normal Distribution

Also Multidimensional Normal Distribution · Multivariate Gaussian · Multidimensional Gaussian · Gaussian Random Vector · Mehrdimensionale Normalverteilung · Multivariate Normalverteilung

A random vector X ∈ ℝⁿ is normally distributed, X ~ N(μ, Σ), if its PDF is p(x) = exp(−½ (x − μ)ᵀ Σ⁻¹ (x − μ)) / √det(2πΣ); then E{X} = μ and cov(X) = Σ.

Know these firstNormal Distribution Covariance Matrix Positive Semidefinite Matrix

Properties#

Moments: $\mathbb{E}\{X\} = \mu$ and $\operatorname{cov}(X) = \Sigma$ (by integral computations), so the mean and the covariance matrix determine the distribution.
$\Sigma$ must be positive definite for the PDF to exist: $\Sigma^{-1}$ appears in the exponent and $\det(2\pi\Sigma) = (2\pi)^n \det\Sigma$ must be positive (Positive Semidefinite Matrix).
Scalar case: for $n = 1$ , $\Sigma = \sigma^2$ and $\det(2\pi\sigma^2) = 2\pi\sigma^2$ , which gives the scalar Normal Distribution.
Linear transformation: for $Y = QX$ with $Q \in \mathbb{R}^{m \times n}$ , $\mu_Y = Q\mu_X, \qquad \Sigma_Y = Q\Sigma_X Q^\top$ (derivation in Covariance Matrix > Properties). If $m = n$ and $Q$ is orthogonal, $\Sigma_Y^{-1} = Q\Sigma_X^{-1}Q^\top$ .
Use in MSI: the least squares estimate with i.i.d. noise of variance $\sigma_\epsilon^2$ is approximately $\hat\theta_\text{LS} \sim \mathcal{N}(\theta_\text{true}, \sigma_\epsilon^2(\Phi^\top\Phi)^{-1})$ for many measurements (Least Squares Estimator).

Example#

ExampleDiagonal covariance

Let $\Sigma = \operatorname{diag}(\sigma_1^2, \sigma_2^2)$ . Then $\det(2\pi\Sigma) = (2\pi)^2\sigma_1^2\sigma_2^2$ and $(x - \mu)^\top\Sigma^{-1}(x - \mu) = \frac{(x_1 - \mu_1)^2}{\sigma_1^2} + \frac{(x_2 - \mu_2)^2}{\sigma_2^2}$ , so

p(x) = \frac{1}{\sqrt{2\pi\sigma_1^2}} e^{-\frac{(x_1-\mu_1)^2}{2\sigma_1^2}} \cdot \frac{1}{\sqrt{2\pi\sigma_2^2}} e^{-\frac{(x_2-\mu_2)^2}{2\sigma_2^2}}.

The PDF factorizes into two scalar normal PDFs, so the components are independent.

Sources#

MSI script pp. 16–17 (§2.3.2, definition and linear transformation example), MSI Slides 02 > 2.3.2 Multidimensional Normal Distribution (not published)
MSI script p. 38 (distribution of the LS estimate)

Connections

Mentioned in inline links6

Least Squares EstimatorMSI
Noise only on the measured outputs: with i.i.d. noise $\epsilon_k$ of variance $\sigma_\epsilon^2$ and exactly known $\phi_k$ , $\hat\theta_\text{LS}$ is unbiased with covariance $\sigma_\epsilon^2(\Phi^\top\Phi)^{-1}$ , and for large $N$ ap… §
DeterminantLA
Use in MSI: normalization $1/\sqrt{\det(2\pi\Sigma)}$ of the Multivariate Normal Distribution; invertibility tests for $\Phi^\top\Phi$ and Jacobians (Invertible Matrix, Jacobian Matrix) §
Covariance Matrix
Symmetric and positive semidefinite: $\Sigma_X = \Sigma_X^\top$ and $z^\top\Sigma_X z = \mathbb{E}\{(z^\top(X - \mu_X))^2\} = \operatorname{Var}(z^\top X) \ge 0$ for all $z$ (Positive Semidefinite Matrix). The Multivariate Normal Distribut… §
Normal Distribution
Vector case: the Multivariate Normal Distribution $\mathcal{N}(\mu, \Sigma)$ replaces $\sigma^2$ by a covariance matrix $\Sigma$ ; for $n = 1$ it reduces to this PDF. §
Moments of Random Variables
Instead of a whole density, one usually works with a few numbers: the mean (where the variable lies), the variance (how far it spreads), the covariance and correlation (how two variables move together), and for random vectors the covarianc… §
Multivariate Normal Distribution — the distribution that is fully described by mean and covariance matrix §
Random Variables and Distributions
A random variable models a quantity that is not known before the experiment, such as a noisy measurement. It is described by probabilities of events, or more conveniently by a PDF. Joint, marginal and conditional densities describe several… §
Multivariate Normal Distribution — the Gaussian for random vectors, fixed by mean vector and covariance matrix §

Search in these mentions →