ConceptDBISAI-written · not yet proofread

Embedding

Also Embeddings

A (word) embedding assigns to each element of a vocabulary V a dense vector in R^d, usually with d much smaller than |V|, such that semantically related words get similar vectors (high cosine similarity).

Know these firstOne-Hot Encoding Cosine Similarity

Definition

Let $V$ be a vocabulary (the set of distinct words). A word embedding is a map $V \to \mathbb{R}^d$ , $w \mapsto \text{emb}(w)$ , that differs from one-hot encoding in two ways:

The dimension $d$ need not be $|V|$ ; typically $d \ll |V|$ , and the vectors are dense (almost all entries nonzero).
The vectors are not pairwise orthogonal: in a good embedding, semantically related words have similar vectors (high cosine similarity), unrelated words do not.

A text (document, chunk) embedding is a single vector in $\mathbb{R}^d$ for a whole piece of text. The simplest one is the sum of the word embeddings of its words: $\text{emb}(w_1 \dots w_k) = \sum_{i=1}^k \text{emb}(w_i)$ .

Properties#

Property	One-Hot Encoding	Word embedding
Dimension	$\lvert V \rvert$ (very large)	$d$ , e.g. 300 (ES11: $d = 300$ , $\lvert V \rvert = 129\,603$ )
Density	sparse (one nonzero entry)	dense
cos-sim of distinct words	always $0$	high for related words

Semantics in the direction: relatedness is encoded in the direction of the vectors, which is exactly what cos-sim compares; its length-invariance also means a summed document embedding (whose length grows with the number of words) is compared by what it is about, not by how long it is.
How embeddings are learned is not covered in DBIS (see IR WS 22/23, Lecture 9); for ES11 they are provided as a dictionary word → tensor.
Sum = bag of words: the summed text embedding, like the VSM document vector (sum of one-hot encodings), ignores word order because addition is commutative: “cute cats eating” and “eating cute cats” get the same vector and cos-sim $1$ .
Sum vs. embedding model: Lecture 12 calls the sum “simplistic”, because not every word is equally relevant for a text. ES12 uses a dedicated embedding API (mixedbread-ai/mxbai-embed-large-v1 at https://ad-llm.cs.uni-freiburg.de/v1/embeddings) that returns one embedding per input text; the same API must be used for chunks and queries.
Uses: similarity search over words or documents (ES11) and retrieving the chunks most similar to a query in Retrieval-Augmented Generation (ES12).

torch.cosine_similarity needs dim=0 for vectors

embs = torch.load("embeddings.pt")          # dict: word -> 1D tensor
sim = torch.cosine_similarity(embs["internet"], embs["web"], dim=0)
doc_emb = embs["internet"] + embs["web"] + embs["surfing"]

Without dim=0 the call fails for 1D tensors.

Example#

ExampleIllustrative 3D embeddings (not from the lecture)

Word	Embedding
internet	$[0.8, 0.6, 0.1]$
web	$[0.7, 0.7, 0.1]$
beach	$[0.1, 0.2, 0.9]$

$\text{cos-sim}(\text{internet}, \text{web}) = \frac{0.99}{\sqrt{1.01}\sqrt{0.99}} \approx 0.99$ , $\;\text{cos-sim}(\text{internet}, \text{beach}) = \frac{0.29}{\sqrt{1.01}\sqrt{0.86}} \approx 0.31$ . With one-hot encodings both values would be $0$ . The document “internet web” has embedding $[1.5, 1.3, 0.2]$ , the same as “web internet”.

Sources#

Slides 11, p. 20 (one-hot has no semantics), Slides 11, p. 21 (definition), Slides 11, p. 22 (embeddings in PyTorch, document embedding as sum)
Slides 12, p. 15 (chunk embeddings), Slides 12, p. 16 (embedding API)
DBIS_VL11 > Word embeddings, DBIS_VL11 > Using the embeddings for ES11, DBIS_VL12 > Computing the embeddings
Exam: DBIS Exam WS2324 (not published) Task 5.1 (cos-sim of two embeddings), DBIS Exam WS2324-2 (not published) Task 5.2 (next-word model on embeddings)

Connections

Mentioned in inline links6

Language Model
A neural language model has better generalization to unseen sequences than counting, because similar contexts get similar internal representations (embeddings). Its last layer typically turns one score per token into the next-token distrib… §
One-Hot Encoding
No semantics: “internet” and “web” are exactly as (dis)similar as “internet” and “beach” (cos-sim $0$ in both cases). This motivates word embeddings: dense, low-dimensional vectors in which related words have high cos-sim. §
Retrieval-Augmented Generation
Embedding: compute an embedding $e_i \in \mathbb{R}^d$ of each chunk $i$ . §
VL11 – Vector Space Model, Word Embeddings
A word embedding also assigns a vector to each element of a vocabulary $V$ , with two differences to one-hot encoding (Slide 21): §
Embeddings — dense, low-dimensional word vectors with semantic cos-sim; document embedding as sum §
VL12 – N-gram Models, LLMs, RAG
Embedding: compute an embedding of each chunk, of the kind seen in Lecture 11 and ES11. ES12: remove the timestamps before embedding; they carry no useful information for this. §
Embeddings — chunk and query embeddings for RAG; sum of word embeddings vs. embedding API §
Word Embeddings and Language Models
Embedding — dense low-dimensional word/text vectors whose cos-sim reflects meaning §
| word embedding (ES11) | learned, dense | sum of word embeddings | $d \ll \lvert V \rvert$ (ES11: 300) | high cos-sim | lost (bag of words) | §

Search in these mentions →