Random Variables and Distributions

Concepts#

1. 1 Random Variablethe object: a value-to-be, with probabilities of events start here
2. 2 Probability Density Functionthe description of a continuous random variable; joint, marginal, conditional
3. 3 Statistical Independencewhen joint probabilities and densities factorize
4. 4 Conditional Probabilityprobabilities and densities given an observed event or value
5. 5 Bayes' Theoremturning $P(B|A)$ into $P(A|B)$: prior to posterior
6. 6 Uniform Distributionthe simplest PDF, constant on an interval
7. 7 Normal Distributionthe Gaussian, fixed by mean and variance
8. 8 Multivariate Normal Distributionthe Gaussian for random vectors, fixed by mean vector and covariance matrix

How it fits together#

Events and densities say the same things at two levels. Every rule for events has a density counterpart, with $P(\cdot)$ replaced by $p(\cdot)$ and sums by integrals:

From the joint density everything else follows. The joint PDF determines the marginals (integrate out), the conditionals (divide by a marginal), and whether the variables are independent (does it factorize?). Bayes’ theorem is not an extra axiom: it is the product rule $p(x, y) = p(x|y)p(y) = p(y|x)p(x)$ solved for $p(x|y)$.

Independence is a statement about the joint density, and it is stronger than it looks. It implies that conditioning changes nothing ($p(x|y) = p(x)$) and that expectations of products factorize, which is the step that makes noise cross terms vanish in estimator analyses (Moments of Random Variables). Mutually exclusive events are the opposite of independent ones: knowing one occurred rules the other out.

Scalar → vector. The Multivariate Normal Distribution has the same structure as the scalar one, with $\sigma^2$ replaced by $\Sigma$:

In both cases the distribution is completely characterized by its first two moments. A diagonal $\Sigma$ makes the density factorize into scalar normals, i.e. independent components.

Uniform vs. normal. Both have a mean and a variance, but the uniform distribution is bounded (all mass on $[a, b]$, variance $(b-a)^2/12$), while the normal one has unbounded support. For estimators this matters: with normal current noise a measured current can be (close to) zero, which is why the Simple Approach Estimator in MSI may not converge.

Notation conventions in MSI. Uppercase $X$ for the random variable, lowercase $x$ for its values, and the index on $p_X$, are dropped from script §2.4 on: $x$ denotes both and $p(x)$ the density.

Key Formulas#

• $P(X \in [a, b]) = \int_a^b p_X(x)\,\mathrm{d}x$ (Probability Density Function)
• $P(A|B) = \frac{P(B|A)P(A)}{P(B)}$ (Bayes’ Theorem)
• $p(x) = \frac{1}{\sqrt{\det(2\pi\Sigma)}}\exp\left(-\frac12(x - \mu)^\top\Sigma^{-1}(x - \mu)\right)$ (Multivariate Normal Distribution)

Lectures#

• No write-up yet. MSI covers this material in script §2.1–2.3 (MSI Slides 02 (not published)), planned for the lectures of 2026-10-21 and 2026-10-28.

Related Topics#

• Moments of Random Variables — mean, variance and covariance summarize these distributions
• Point Estimation — estimates are random variables with these distributions