ConceptDBISAI-written · not yet proofread

One-Hot Encoding

Given an ordered vocabulary V, the one-hot encoding of a word is the vector in {0,1}^|V| that is 1 at the word's index and 0 everywhere else.

Properties#

Pairwise orthogonal: $e_i \cdot e_j = 0$ for $i \neq j$ , hence $\text{cos-sim}(e_i, e_j) = 0$ (Cosine Similarity).
Documents and queries are sums: a document column of the term-document matrix in the Vector Space Model (VSM) is the sum of the one-hot encodings of its words, with multiplicity: $d = \sum_{w \in \text{doc}} e_w$ (with tf entries). A query vector is the same sum; since a query typically contains few keywords, each at most once, it has a few 1s and 0s otherwise.
Dimension $|V|$ : very large for real vocabularies, and the vectors are maximally sparse (one nonzero entry).
No semantics: “internet” and “web” are exactly as (dis)similar as “internet” and “beach” (cos-sim $0$ in both cases). This motivates word embeddings: dense, low-dimensional vectors in which related words have high cos-sim.

Example $V = (\text{internet}, \text{web}, \text{surfing}, \text{beach})$

Document “internet web surfing beach surfing”:

[1,0,0,0] + [0,1,0,0] + [0,0,1,0] + [0,0,0,1] + [0,0,1,0] = [1, 1, 2, 1],

which is column D4 of the lecture’s term-document matrix.

Connections

Embedding
Let $V$ be a vocabulary (the set of distinct words). A word embedding is a map $V \to \mathbb{R}^d$ , $w \mapsto \text{emb}(w)$ , that differs from one-hot encoding in two ways: §
| Property | One-Hot Encoding | Word embedding | §
VL11 – Vector Space Model, Word Embeddings
Given an ordered vocabulary $V$ , word $w$ is represented by the vector $[0, \dots, 0, 1, 0, \dots, 0] \in \{0,1\}^{|V|}$ that is $1$ at the index of $w$ in $V$ and $0$ everywhere else. This works for any ordered set, not just words (Slide… §
One-Hot Encoding — words as orthogonal unit vectors; documents are sums of one-hot encodings §
Word Embeddings and Language Models
One-Hot Encoding — words as orthogonal unit vectors; a VSM column is their sum §
| VSM / one-hot | $e_i$ , orthogonal | sum of one-hot encodings = column of $A$ (tf), or BM25 scores | $\lvert V \rvert$ , sparse | cos-sim $0$ | lost (bag of words) | §