ConceptDBISAI-written · not yet proofread

Softmax

The softmax function maps a vector z ∈ ℝ^k to the probability distribution with components e^(z_i) / Σ_j e^(z_j).

Know these firstSigmoid

Properties#

$\text{softmax}(z)_i \in (0, 1)$ for all $i$ , and $\sum_i \text{softmax}(z)_i = 1$ .
Preserves ordering: if $z_i > z_j$ then $\text{softmax}(z)_i > \text{softmax}(z)_j$ . So $\arg\max_i \text{softmax}(z)_i = \arg\max_i z_i$ .
Invariant under constant shift: $\text{softmax}(z + c) = \text{softmax}(z)$ for any scalar $c$ (the factor $e^c$ cancels). Implementations subtract $\max_i z_i$ before exponentiating to avoid overflow.
Relation to Sigmoid (two-class case, exam WS 23/24, Task 5.3): $\text{softmax}([x, 0])_1 = \frac{e^x}{e^x + 1} = \frac{1}{1 + e^{-x}} = \sigma(x)$ $\text{softmax}([x, 0])_2 = \frac{1}{e^x + 1} = \sigma(-x)$ By shift invariance, any two scores $(z_1, z_2)$ give $\text{softmax}(z)_1 = \sigma(z_1 - z_2)$ . Multi-class logistic regression with softmax over $k = 2$ scores is therefore ordinary Logistic Regression.

Example#

$\text{softmax}([2, 1, 0]) = \left(\frac{e^2}{e^2 + e + 1},\; \frac{e}{e^2 + e + 1},\; \frac{1}{e^2 + e + 1}\right) \approx (0.665, 0.245, 0.090)$

Shifting by $-2$ gives the same result: $\text{softmax}([0, -1, -2]) = \frac{(1, e^{-1}, e^{-2})}{1 + e^{-1} + e^{-2}} \approx (0.665, 0.245, 0.090)$ .

Sources#

Not covered by a WS 25/26 lecture. Source: the DBIS exams.
Exam WS 23/24, Task 5.3 (not published) (softmax of $(x, 0)$ in terms of $\sigma$ )

Connections

Prerequisites know first1

Sigmoid

Mentioned in inline links6

Language Model
A neural language model has better generalization to unseen sequences than counting, because similar contexts get similar internal representations (embeddings). Its last layer typically turns one score per token into the next-token distrib… §
Large Language Model
LLMs have billions of parameters. Temperature $T$ divides the scores before the softmax, $\Pr(v) \propto e^{z_v / T}$ : $T \to 0$ approaches always taking the most likely token, large $T$ approaches uniform sampling. §
Logistic Regression
Multi-class generalization: replace $\sigma$ by Softmax over one score per class. For two classes with scores $(x, 0)$ , softmax gives $(\sigma(x), \sigma(-x))$ , i.e. exactly logistic regression. §
Sigmoid
Special case of Softmax for two classes: $\sigma(x) = \text{softmax}([x, 0])_1$ . §
Linear Classifiers and Logistic Regression
Softmax — vector to probability distribution; two-class case equals the sigmoid §
Softmax: $e^{z_i}/\sum_j e^{z_j}$ (Softmax) §
Word Embeddings and Language Models
Softmax — turns a score vector into a probability distribution (exam: relation to sigmoid) §
Not lectured in WS 25/26, but asked in WS 23/24 Task 5.3 under this topic: softmax maps a score vector $z \in \mathbb{R}^k$ to a distribution, $\text{softmax}(z)_i = e^{z_i} / \sum_j e^{z_j}$ , and the sigmoid $\sigma(z) = 1 / (1 + e^{-z})$ … §

Search in these mentions →