ConceptDBISAI-written · not yet proofread

Sigmoid

The sigmoid (logistic) function σ(z) = 1 / (1 + e^(−z)) maps every real number strictly increasingly into the interval (0, 1).

Formula#

Derivative:

\sigma'(z) = \sigma(z) \cdot (1 - \sigma(z)) = \sigma(z) \cdot \sigma(-z)

Symmetry:

\sigma(-z) = 1 - \sigma(z)

ProofProof of

\sigma'(t) = \sigma(t)\,\sigma(-t)

(exam WS 21/22, Task 5.3)

Chain rule: $\sigma'(t) = \frac{-1}{(1 + e^{-t})^2} \cdot (-e^{-t}) = \frac{e^{-t}}{(1 + e^{-t})^2} = \frac{1}{1 + e^{-t}} \cdot \frac{e^{-t}}{1 + e^{-t}}$ .
The first factor is $\sigma(t)$ .
Second factor, multiplied by $e^{t}/e^{t}$ : $\frac{e^{-t}}{1 + e^{-t}} = \frac{1}{e^{t} + 1} = \sigma(-t)$ .

Hence $\sigma'(t) = \sigma(t)\,\sigma(-t)$ . The symmetry follows from $\sigma(z) + \sigma(-z) = \frac{e^z}{e^z + 1} + \frac{1}{e^z + 1} = 1$ .

Properties#

$\sigma(0) = 0.5$
$\lim_{z \to \infty} \sigma(z) = 1$ , $\lim_{z \to -\infty} \sigma(z) = 0$
Monotonically (strictly) increasing, since $\sigma'(z) = \sigma(z)\sigma(-z) > 0$ . Hence $\sigma(z) > 1/2 \iff z > 0$ .
Special case of Softmax for two classes: $\sigma(x) = \text{softmax}([x, 0])_1$ .
Used as the output function of Logistic Regression, where $\sigma(w \cdot x)$ is the predicted probability of class 1. The derivative formula gives the simple gradient $(\sigma(w \cdot x) - y)\,x$ of the cross-entropy loss.

Example#

$\sigma(0) = 0.5$ , $\sigma(1) \approx 0.731$ , $\sigma(-1) \approx 0.269$ , $\sigma(2) \approx 0.881$ . Check of the symmetry: $\sigma(1) + \sigma(-1) = 1$ . Check of the derivative: $\sigma'(0) = \sigma(0)\,\sigma(0) = 0.25$ , the maximal slope.

Sources#

Not covered by a WS 25/26 lecture. Source: the DBIS/IR exams.
Exam WS 21/22, Task 5.3 (not published) (derivative)
Exam WS 23/24, Task 5.2–5.3 (not published) (torch.sigmoid in training, softmax of $(x, 0)$ )
Exam WS 23/24 (2nd date), Task 5.1 (not published) (definition in the logistic regression model)

Connections

Mentioned in inline links5

Linear Classifier
Logistic Regression predicts the positive class iff $\sigma(w \cdot x) > 1/2 \iff w \cdot x > 0$ (Sigmoid); §
Logistic Regression
(Sigmoid), and uses the binary cross-entropy loss between a label $y \in [0, 1]$ (usually $y \in \{0, 1\}$ ) and a prediction $y' \in (0, 1)$ : §
Use $\sigma'(z) = \sigma(z)\,\sigma(-z)$ and $1 - \sigma(z) = \sigma(-z)$ (Sigmoid). Then §
Softmax
Relation to Sigmoid (two-class case, exam WS 23/24, Task 5.3): §
Linear Classifiers and Logistic Regression
Sigmoid — $\sigma(z) = 1/(1 + e^{-z})$ , derivative $\sigma(z)\sigma(-z)$ §
Chain for logistic regression: score $z = w \cdot x$ $\to$ Sigmoid gives a probability $\to$ cross-entropy compares it with the label $\to$ $\sigma' = \sigma(z)\sigma(-z)$ makes the gradient collapse to $(\sigma(z) - y)\,x$ $\to$ gradient… §
Word Embeddings and Language Models
Sigmoid — two-class special case of softmax (WS 23/24 Task 5.3) §
Not lectured in WS 25/26, but asked in WS 23/24 Task 5.3 under this topic: softmax maps a score vector $z \in \mathbb{R}^k$ to a distribution, $\text{softmax}(z)_i = e^{z_i} / \sum_j e^{z_j}$ , and the sigmoid $\sigma(z) = 1 / (1 + e^{-z})$ … §

Search in these mentions →