TopicLinear Algebra used inDBISAI-written · not yet proofread

Inner Products and Norms

How the dot product induces lengths, distances and angles in $R^n$, and how the angle gives the length-independent cosine similarity used to compare vectors.

Concepts#

Cosine Similaritythe cosine of the angle between two vectors: dot product normalized by both lengths start here

How it fits together#

Chain. Dot product → norm → distance and angle:

Quantity	Formula	Measures	Under $x \mapsto \lambda x$ ( $\lambda > 0$ )
dot product	$x \cdot y = \sum_i x_i y_i$	alignment and lengths	scales by $\lambda$
Euclidean norm	$\lVert x \rVert_2 = \sqrt{x \cdot x}$	length	scales by $\lambda$
Euclidean distance	$d(x, y) = \lVert x - y \rVert_2$	dissimilarity (small = close)	changes
cosine similarity	$\dfrac{x \cdot y}{\lVert x \rVert_2\lVert y \rVert_2} = \cos\omega$	direction only (large = similar)	unchanged

Cauchy–Schwarz makes the angle well-defined: $|x \cdot y| \le \|x\|_2\|y\|_2$ , so the quotient lies in $[-1, 1]$ and there is a unique $\omega \in [0, \pi]$ with that cosine.
Orthogonality is the case $x \cdot y = 0$ ( $\omega = 90°$ ). Orthogonal unit vectors are orthonormal; the standard basis vectors $e_1, \dots, e_n$ are the typical example.
Normalizing turns cosine into dot product: for unit vectors, $\text{cos-sim}(x, y) = x \cdot y$ and $\|x - y\|_2^2 = 2 - 2\,x \cdot y$ . On the unit sphere, “highest cosine similarity”, “highest dot product” and “smallest Euclidean distance” give the same ranking.
Which to use: the dot product when length carries meaning (e.g. more occurrences = more relevant), cosine similarity when only the direction matters (e.g. documents of different length), Euclidean distance when absolute position matters.
Matrix form: the dot products of a vector $q$ with all columns $a_1, \dots, a_n$ of $A$ are the entries of $q^\top A$ ; all pairwise dot products of the columns form the Gram matrix $A^\top A$ , which is positive semidefinite because $x^\top A^\top A x = \|Ax\|_2^2 \ge 0$ .

Key Formulas#

$\|x\|_2 = \sqrt{x \cdot x}$ , $\;|x \cdot y| \le \|x\|_2\|y\|_2$ (Cauchy–Schwarz)
$\text{cos-sim}(x, y) = \dfrac{x \cdot y}{\|x\|_2\|y\|_2} = \cos\omega$ (Cosine Similarity)
$\text{cos-sim}(x, y) = x' \cdot y'$ with $x' = x / \|x\|_2$ , $y' = y / \|y\|_2$ (Cosine Similarity > Formula)

Lectures#

No lecture yet. Source: MML ch. 3.1–3.4, pp. 71–77.
Used in DBIS: DBIS_VL11 (cosine similarity for document ranking, word embeddings), DBIS_VL12 (retrieving RAG chunks), see Word Embeddings and Language Models.

Rank and Invertibility — the Gram matrix $A^\top A$ of all column dot products is PSD and invertible iff $A$ has full column rank