ConceptDBISAI-written · not yet proofread

Logistic Regression

A model for functions from ℝ^n to [0, 1] that predicts σ(w·x) with the sigmoid σ and learns the weights w by minimizing the binary cross-entropy loss, e.g. with the update w ← w + η·(y − σ(w·x))·x per example.

Know these firstLinear Classifier Sigmoid

Definition

For a function $f : X \to Y$ , a model with $N$ parameters is a function $M : X \times \mathbb{R}^N \to Y$ and a loss function is $L : Y \times Y \to \mathbb{R}$ . Logistic regression models $f : \mathbb{R}^n \to [0, 1]$ with $N = n$ parameters $w \in \mathbb{R}^n$ by

M(x, w) = \sigma(w \cdot x), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}

(Sigmoid), and uses the binary cross-entropy loss between a label $y \in [0, 1]$ (usually $y \in \{0, 1\}$ ) and a prediction $y' \in (0, 1)$ :

L(y, y') = -\big[\,y \ln y' + (1 - y) \ln (1 - y')\,\big].

Training chooses $w$ to minimize the summed loss $\sum_i L(y_i, M(x_i, w))$ over the training examples $(x_i, y_i)$ .

Formula#

Gradient of the loss for one example $(x, y)$ with $z = w \cdot x$ :

\nabla_w L\big(y, \sigma(w \cdot x)\big) = \big(\sigma(w \cdot x) - y\big) \cdot x.

ProofDerivation

Use $\sigma'(z) = \sigma(z)\,\sigma(-z)$ and $1 - \sigma(z) = \sigma(-z)$ (Sigmoid > Formula). Then

\frac{\partial L}{\partial z} = -\Big[y \frac{\sigma'(z)}{\sigma(z)} - (1 - y)\frac{\sigma'(z)}{1 - \sigma(z)}\Big] = -\big[y\,\sigma(-z) - (1 - y)\,\sigma(z)\big] = -\big[y - y\sigma(z) - \sigma(z) + y\sigma(z)\big] = \sigma(z) - y,

and by the chain rule $\nabla_w L = \frac{\partial L}{\partial z} \cdot \nabla_w z = (\sigma(z) - y)\,x$ .

Stochastic gradient descent with batch size 1 and learning rate $\eta$ : for each training example in turn,

w \leftarrow w + \eta \cdot \big(y_i - \sigma(w \cdot x_i)\big) \cdot x_i.

One pass over all examples is an epoch. The factor $y_i - \sigma(w \cdot x_i)$ is the error between label and predicted probability.

Bias. A bias is included by a constant component $1$ in every $x$ (then $w$ contains the bias weight), see Linear Classifier > Formula.

Properties#

It is a Linear Classifier: predicting class 1 iff $\sigma(w \cdot x) > 1/2$ is the same as $w \cdot x > 0$ , because $\sigma$ is strictly increasing with $\sigma(0) = 1/2$ . Unlike the Perceptron, it also outputs a probability, and the update is proportional to the error instead of only happening on mistakes.
Loss behaviour: for $y = 1$ the loss is $-\ln y'$ , which is $0$ at $y' = 1$ and grows to $\infty$ as $y' \to 0$ ; symmetrically for $y = 0$ . Confidently wrong predictions are punished heavily.
Multi-class generalization: replace $\sigma$ by Softmax over one score per class. For two classes with scores $(x, 0)$ , softmax gives $(\sigma(x), \sigma(-x))$ , i.e. exactly logistic regression.

Example#

def epoch(X: torch.Tensor, y: torch.Tensor, w: torch.Tensor) -> torch.Tensor:
    for i in range(X.shape[0]):
        factor = 0.1 * (y[i] - torch.sigmoid(torch.dot(X[i], w)))
        w = w + factor * X[i]
    return w

(Exam WS 23/24, Task 5.2: one epoch, batch size 1, learning rate 0.1, bias already in X.)

Sources#

Not covered by a WS 25/26 lecture. Source: the DBIS/IR exams.
Exam WS 23/24 (2nd date), Task 5.1 (not published) (model and loss)
Exam WS 23/24, Task 5.2 (not published) (one training epoch)

Connections

Mentioned in inline links4

Linear Classifier
Bias as an extra component. Appending a constant $1$ to $x$ and $-b$ to $w$ gives $w \cdot x - b = (w, -b) \cdot (x, 1)$ , a hyperplane through the origin in $\mathbb{R}^{d+1}$ . Logistic Regression uses this form (“bias term already include… §
Logistic Regression predicts the positive class iff $\sigma(w \cdot x) > 1/2 \iff w \cdot x > 0$ (Sigmoid); §
Sigmoid
Used as the output function of Logistic Regression, where $\sigma(w \cdot x)$ is the predicted probability of class 1. The derivative formula gives the simple gradient $(\sigma(w \cdot x) - y)\,x$ of the cross-entropy loss. §
Softmax
By shift invariance, any two scores $(z_1, z_2)$ give $\text{softmax}(z)_1 = \sigma(z_1 - z_2)$ . Multi-class logistic regression with softmax over $k = 2$ scores is therefore ordinary Logistic Regression. §
Linear Classifiers and Logistic Regression
Logistic Regression — model $\sigma(w \cdot x)$ , binary cross-entropy, update $w \leftarrow w + \eta (y - \sigma(w \cdot x)) x$ §
| Logistic Regression | gradient of cross-entropy | probability $\sigma(w \cdot x)$ | on every example, scaled by the error | §

Search in these mentions →