ConceptProbability and Statistics used inMSIAI-written · not yet proofread

Statistical Independence

Also Independence · Independent Random Variables · Independent Events · Stochastic Independence · Dependent · Stochastische Unabhängigkeit · Unabhängigkeit

Two events are independent if the probability that both occur is the product of their probabilities; two random variables are independent if their joint PDF is the product of their individual PDFs.

Know these firstRandom Variable Probability Density Function

Properties#

Conditioning changes nothing: for independent $X, Y$ , the conditional PDF is $p_{X|Y}(x|y) = \frac{p_X(x)\,p_Y(y)}{p_Y(y)} = p_X(x)$ (Conditional Probability); likewise $P(A|B) = P(A)$ for independent events with $P(B) > 0$ .
Expectation of a product: for independent $X, Y$ , $\mathbb{E}\{XY\} = \mathbb{E}\{X\}\,\mathbb{E}\{Y\}$ (the double integral of $xy\,p_X(x)p_Y(y)$ splits). The MSI script uses this when it computes $\mathbb{E}\{y_k y_j\} = \mu_y^2$ for independent measurements $j \ne k$ (Sample Mean).
Independent ⇒ uncorrelated, not conversely: independence implies zero Covariance and Correlation. The converse fails in general (see Correlation > Example).
Mutually exclusive ≠ independent: if $P(A), P(B) > 0$ and $A$ , $B$ never occur together, then $P(A \wedge B) = 0 \ne P(A)P(B)$ , so the events are dependent.

Example#

Sources#

MSI script p. 13 (events, §2.1), MSI Slides 02 > 2.1 Random Variables and Probabilities (not published)
MSI script p. 14 (random variables, §2.2), p. 16 (independent ⇒ uncorrelated)
MSI script pp. 19–20 (§2.5, noise assumptions), MSI Slides 02 > 2.5 Analysis of the Resistance Estimation Example (not published)

Connections

Mentioned in inline links14

Hidden Markov ModelDBIS
The factorization follows from two independence assumptions (Conditional Probability, Statistical Independence): §
Naive Bayes ClassifierDBIS
Derivation. By Bayes’ Theorem, $\Pr(c \mid d) = \Pr(d \mid c)\,\Pr(c) / \Pr(d)$ . The denominator $\Pr(d)$ is the same for all classes and drops out of the $\arg\max$ . The naive assumption is that the words of $d$ are independent given the… §
Naive Bayes ClassificationDBIS
Prerequisites from the foundations: Bayes’ Theorem, Conditional Probability, Statistical Independence. §
Chain: Bayes’ Theorem turns $\Pr(c \mid d)$ into $\Pr(d \mid c)\Pr(c)/\Pr(d)$ ; $\Pr(d)$ is constant over classes and dropped; conditional independence of the words factorizes $\Pr(d \mid c)$ into the $p_{wc}$ . §
POS-TaggingDBIS
Prerequisites from the foundations: Conditional Probability, Statistical Independence. §
VL02 – Probability and Statistics in a NutshellMSI
The section introduces a Random Variable $X$ as a description of an experiment’s outcomes. $X$ has no value itself and only takes values $x$ . It then gives the rules for probabilities of events: the sum for mutually exclusive events and th… §
Statistical Independence: product rule for events and joint PDFs §
Conditional Probability
Independence: $A$ , $B$ are independent iff $P(A|B) = P(A)$ ; for densities iff $p(x|y) = p(x)$ . §
Correlation
Independence ⇒ uncorrelated, not conversely: independent variables have $\sigma(Y, Z) = 0$ , hence $\rho = 0$ . Uncorrelated variables can still be dependent (example below). Correlation only measures linear dependence. §
Covariance
Independent ⇒ zero covariance: for independent $Y$ , $Z$ , $\mathbb{E}\{YZ\} = \mu_Y\mu_Z$ , so $\sigma(Y, Z) = 0$ . The converse is false. §
Zero covariance does not mean independence. Left: $Z = Y^2$ with $Y \sim \mathcal{N}(0, 1)$ , so $Z$ is fully determined by $Y$ . But every point $(y, y^2)$ has an equally likely mirror point $(-y, y^2)$ with the opposite product, so the qua… §
Expected Value
Product of independent variables: $\mathbb{E}\{XY\} = \mathbb{E}\{X\}\,\mathbb{E}\{Y\}$ if $X$ , $Y$ are independent. §
Multivariate Normal Distribution
The PDF factorizes into two scalar normal PDFs, so the components are independent. §
Probability Density Function
Independence: $X, Y$ are independent iff $p_{X,Y}(x, y) = p_X(x)\,p_Y(y)$ . §
Random Variable
$A$ , $B$ independent ⇔ $P(A \wedge B) = P(A)P(B)$ . The joint probability $P(A \wedge B)$ is also written $P(A, B)$ . §
Sample Mean
using that the $N$ squares have $\mathbb{E}\{y_k^2\} = \sigma_y^2 + \mu_y^2$ and the $N(N-1)$ cross terms $y_k y_j$ , $j \ne k$ , are products of independent variables. Hence… §
Random Variables and Distributions
A random variable models a quantity that is not known before the experiment, such as a noisy measurement. It is described by probabilities of events, or more conveniently by a PDF. Joint, marginal and conditional densities describe several… §
Statistical Independence — when joint probabilities and densities factorize §

Search in these mentions →