ConceptLinear Algebra used inDBISAI-written · not yet proofread

Cosine Similarity

Also cos-similarity

The cosine similarity of two nonzero vectors is their dot product divided by the product of their Euclidean norms, cos-sim(x, y) = x·y / (‖x‖ ‖y‖), which equals the cosine of the angle between them.

Notation: DBIS writes $x \bullet y$ for the dot product and $|x|$ for $\|x\|_2$ . MML defines the angle for any inner product $\langle \cdot, \cdot \rangle$ as $\cos\omega = \langle x, y \rangle / (\|x\| \|y\|)$ ; cosine similarity is the case of the dot product.

Formula#

Normalize first. With the unit vectors $x' = x / \|x\|_2$ and $y' = y / \|y\|_2$ (so $\|x'\|_2 = \|y'\|_2 = 1$ ):

\text{cos-sim}(x, y) = \text{cos-sim}(x', y') = x' \cdot y'.

So after normalizing all vectors once, every cosine similarity is a plain dot product, and the similarities of one vector $q$ to all columns of a matrix $A$ are the single product $q'^\top A'$ ( $A'$ = $A$ with normalized columns).

Properties#

Range: $\text{cos-sim}(x, y) \in [-1, 1]$ $cos-sim (x, y) \in [- 1, 1]$ , by the Cauchy–Schwarz inequality $|x \cdot y| \le \|x\|_2 \|y\|_2$ $∣ x \cdot y ∣ \leq ∥ x ∥_{2} ∥ y ∥_{2}$ .
- $= 1$ : same direction ( $y = \lambda x$ , $\lambda > 0$ ).
- $= 0$ : orthogonal ( $x \cdot y = 0$ ).
- $= -1$ : opposite direction ( $y = \lambda x$ , $\lambda < 0$ ).
- For vectors with nonnegative entries (e.g. term counts) it lies in $[0, 1]$ .
Scale invariance: $\text{cos-sim}(\lambda x, y) = \text{cos-sim}(x, y)$ for every $\lambda > 0$ ; for $\lambda < 0$ the sign flips. It compares directions only and ignores lengths, unlike the plain dot product, which grows with $\|x\|_2$ .
Symmetric: $\text{cos-sim}(x, y) = \text{cos-sim}(y, x)$ .
Not a metric: it is a similarity (large = similar), not a distance. For unit vectors it determines the Euclidean distance: $\|x' - y'\|_2^2 = 2 - 2\,\text{cos-sim}(x, y)$ . So ranking by highest cos-sim is the same as ranking normalized vectors by smallest Euclidean distance.
Undefined for the zero vector (division by $0$ ).

ProofVia the law of cosines

The triangle spanned by $x$ and $y$ has the third side $x - y$ . The law of cosines ( $c^2 = a^2 + b^2 - 2ab\cos\gamma$ , which for $\gamma = 90°$ is Pythagoras) gives

\|x - y\|_2^2 = \|x\|_2^2 + \|y\|_2^2 - 2\|x\|_2\|y\|_2 \cos\omega.

Expanding with the dot product: $\|x - y\|_2^2 = (x - y)\cdot(x - y) = \|x\|_2^2 - 2\,x \cdot y + \|y\|_2^2$ . Comparing both sides gives $x \cdot y = \|x\|_2 \|y\|_2 \cos\omega$ .

Example#

Sources#

MML p. 72 (Euclidean norm, dot product), MML p. 75 (induced norm (3.16), Cauchy–Schwarz (3.17))
MML p. 76 (angle between vectors (3.25)), MML p. 77 (Example 3.6, Def. 3.7 orthogonality)
Use in DBIS: Slides 11, p. 12 (definition), Slides 11, p. 13 (normalize in advance), DBIS_VL11 > Variant 2: cosine similarity, DBIS_VL11 > Efficient computation, DBIS_VL12 > Processing a query; see Word Embeddings and Language Models for document ranking, word embeddings and RAG
Exam: DBIS Exam WS2324 (not published) Task 5.1 (implement cos_sim without numpy/torch)

Connections

Mentioned in inline links8

EmbeddingDBIS
The vectors are not pairwise orthogonal: in a good embedding, semantically related words have similar vectors (high cosine similarity), unrelated words do not. §
One-Hot EncodingDBIS
Pairwise orthogonal: $e_i \cdot e_j = 0$ for $i \neq j$ , hence $\text{cos-sim}(e_i, e_j) = 0$ (Cosine Similarity). §
Retrieval-Augmented GenerationDBIS
Find the $k$ chunks with the highest cosine similarity $\text{cos-sim}(e_q, e_i)$ . §
Retrieval: a linear scan over all chunk embeddings suffices for ES12; real applications precompute a special data structure. With normalized embeddings, all similarities are one matrix-vector product (Cosine Similarity). §
Vector Space ModelDBIS
Another method to measure relevancy is using the cos-similarity. §
VL11 – Vector Space Model, Word EmbeddingsDBIS
where $\bullet$ is the dot product and $|\cdot|_2$ the $L_2$ -norm (Slide 12). See cos-similarity. §
cos-similarity — length-normalized similarity, equal to the cosine of the angle; precompute normalized vectors §
VL12 – N-gram Models, LLMs, RAGDBIS
Find the $k$ chunks whose embeddings have the highest cosine similarity to the query embedding. ES12: a linear scan over all chunks suffices; real applications precompute a special data structure for this. §
cos-similarity — ranks chunks against the query embedding §
Word Embeddings and Language ModelsDBIS
Cosine Similarity — length-normalized dot product; ranks documents, words and chunks by direction §
| cos-sim | any; normalize the columns of $A$ and $q$ once | length removed entirely (direction only) | $q'^\top A'$ | ES11, Ex. 2 | §
Inner Products and Norms
The dot product $x \cdot y = \sum_i x_i y_i$ is the one operation from which the geometry of $\mathbb{R}^n$ follows: the length $\|x\|_2 = \sqrt{x \cdot x}$ , the distance $\|x - y\|_2$ , and the angle $\omega$ with… §
Cosine Similarity — the cosine of the angle between two vectors: dot product normalized by both lengths §

Search in these mentions →