ConceptProbability and Statistics used inMSIAI-written · not yet proofread

Covariance

Also Kovarianz

The covariance σ(Y, Z) = E{(Y − μ_Y)(Z − μ_Z)} of two scalar random variables measures how they vary together; its unit is the product of their units.

Know these firstExpected Value Variance

Formula#

\sigma(Y, Z) = \mathbb{E}\{YZ\} - \mu_Y\mu_Z

(expand the product and use linearity, as for the Variance).

Properties#

Symmetric, and $\sigma(Y, Y) = \sigma_Y^2$ .
Units: the product of the units of $Y$ and $Z$ . Dividing by $\sigma_Y\sigma_Z$ gives the unitless Correlation.
Independent ⇒ zero covariance: for independent $Y$ , $Z$ , $\mathbb{E}\{YZ\} = \mu_Y\mu_Z$ , so $\sigma(Y, Z) = 0$ . The converse is false.
Bilinear: $\sigma(a + bY, c + dZ) = bd\,\sigma(Y, Z)$ .
Vector form: all variances and covariances of a random vector are collected in the Covariance Matrix, with entries $(\Sigma_X)_{jk} = \sigma(X_j, X_k)$ .

Example#

ExampleSample mean and one measurement (MSI script)

Let $y_1, \dots, y_N$ be independent with mean $\mu_y$ and variance $\sigma_y^2$ , and $M(y) = \frac1N\sum_{k=1}^N y_k$ the Sample Mean. Using $\mathbb{E}\{M(y)\} = \mu_y$ :

\operatorname{cov}(M(y), y_k) = \mathbb{E}\{M(y)\,y_k\} - \mu_y^2 = \frac1N\big(\underbrace{\mathbb{E}\{y_k^2\}}_{\sigma_y^2 + \mu_y^2} + (N-1)\mu_y^2\big) - \mu_y^2 = \frac{\sigma_y^2}{N}.

Only the term $y_k \cdot y_k$ contributes; the other $N - 1$ products are independent and give $\mu_y^2$ . A single measurement influences the sample mean less and less as $N$ grows.

Intuition#

Every point

(y, z)

of the plane contributes the product of its offsets from the means,

(y - \mu_Y)(z - \mu_Z)

. The product is positive in the upper-right and lower-left quadrant around

(\mu_Y, \mu_Z)

and negative in the other two (left). The joint PDF weights each contribution by how much probability lies near that point (middle). The covariance is the volume under the weighted product (right): red volume minus blue volume. A joint PDF stretched along the rising diagonal puts most of its mass into the

+

quadrants, so

\sigma(Y, Z) > 0

. Along the falling diagonal,

\sigma(Y, Z) < 0

Looking at the joint PDF from the side gives the PDF of one variable, the marginal

p_Y(y) = \int p_{Y,Z}(y, z)\,\mathrm{d}z

. “From the side” means adding up everything along the line of sight, like an X-ray, not the outline of the surface. For a normal distribution both have the same shape; in general they don’t. The marginals don’t determine the covariance: tilting the surface (changing

\rho

) leaves both marginals unchanged. The covariance lives in the joint PDF only.

N = 400

samples of

(Y, Z)

from a bivariate normal distribution with

\sigma_Y = \sigma_Z = 1

, so

\sigma(Y, Z) = \rho

(Correlation). Red points have a positive offset product, blue ones a negative. The sample covariance

\frac1N \sum_{k=1}^N (y_k - \bar y)(z_k - \bar z)

, with sample means

\bar y

\bar z

, is the same weighting with the PDF replaced by the points: a rising cloud has mostly red points, a falling one mostly blue, and a round cloud about as many of each.

Zero covariance does not mean independence. Left:

Z = Y^2

with

Y \sim \mathcal{N}(0, 1)

, so

Z

is fully determined by

Y

. But every point

(y, y^2)

has an equally likely mirror point

(-y, y^2)

with the opposite product, so the quadrants cancel:

\sigma(Y, Z) = \mathbb{E}\{Y \cdot Y^2\} - \mu_Y\mu_Z = \mathbb{E}\{Y^3\} - 0 = 0

. Right: points on a circle,

\sigma(Y, Z) = 0

by symmetry. Covariance only measures how much the joint PDF leans along a straight line.

Sources#

MSI script pp. 15–16 (§2.2.4), MSI Slides 02 > 2.2.4 Covariance and Correlation (not published)
MSI script p. 17 (example: covariance of the sample mean and a single measurement), MSI Slides 02 > 2.4 Statistical Estimators (not published)

Connections

Mentioned in inline links8

VL02 – Probability and Statistics in a NutshellMSI
Marginal PDFs come from integrating the joint PDF. The Covariance $\sigma(Y,Z)$ measures how $Y$ and $Z$ vary together, in the product of their units. The Correlation $\rho(Y,Z) = \sigma(Y,Z)/(\sigma_Y\sigma_Z)$ is its unitless version in… §
Covariance: joint variation of two variables §
Correlation
The correlation of two scalar random variables $Y$ , $Z$ with standard deviations $\sigma_Y, \sigma_Z > 0$ is their covariance divided by both standard deviations, §
Correlation is covariance with the scale removed. Multiplying $Z$ by $b > 0$ (e.g. switching from meters to centimeters) multiplies every offset product $(y - \mu_Y)(z - \mu_Z)$ , and with it $\sigma(Y, Z)$ , by $b$ (top row). The standard d… §
Covariance Matrix
Its entries are $(\Sigma_X)_{jk} = \sigma(X_j, X_k)$ (Covariance); the diagonal holds the variances $\sigma_{X_j}^2$ . §
Expected Value
Higher moments: Variance, Covariance and the Covariance Matrix are all expectations of products of centred variables $X - \mu_X$ . §
Sample Mean
Covariance with one measurement: $\operatorname{cov}(M(y), y_k) = \frac{\sigma_y^2}{N}$ , so $\mathbb{E}\{y_k M(y)\} = \frac{\sigma_y^2}{N} + \mu_y^2$ (derivation in Covariance). §
Statistical Independence
Independent ⇒ uncorrelated, not conversely: independence implies zero Covariance and Correlation. The converse fails in general (see Correlation). §
Variance
Generalizations: the variance is the Covariance of a variable with itself, $\sigma(X, X) = \sigma_X^2$ , and the diagonal of the Covariance Matrix. §
Moments of Random Variables
Instead of a whole density, one usually works with a few numbers: the mean (where the variable lies), the variance (how far it spreads), the covariance and correlation (how two variables move together), and for random vectors the covarianc… §
Covariance — the joint spread of two variables §

Search in these mentions →