Linear Classifiers and Logistic Regression

4.1: after object 1 (boundary): $w = -1$, $b = 1$; objects 2 to 4 are correct; object 5: $w = 4$, $b = 0$; objects 6 to 8 correct. End of round 1: $w = 4$, $b = 0$. 4.2: see the code in Perceptron > Example; the result is $w^* = 5$, $b^* = 23$. 4.3: induction over the update steps; if $(\bar w_i, \bar b_i) = (-w_i, -b_i)$ then the test $(-y)(\bar w_i x - \bar b_i) \le 0$ is the same as $y(w_i x - b_i) \le 0$, so both runs update at the same steps, and $\bar w_{i+1} = -w_i - yx = -w_{i+1}$, $\bar b_{i+1} = -b_i + y = -b_{i+1}$. The base case is $0 = -0$. For the swapped labels the simulation gives $(-5, -23)$.

The normal vector is $w = (2, -2, 1)$ with $|w| = \sqrt{4 + 4 + 1} = 3$. $\text{dist}(x, H) = \frac{|w \cdot x - b|}{|w|}$ with $w \cdot x = 2 - 4 + 3 = 1$, so $\frac{|1 - 7|}{3} = 2$.

1. Chain rule: $\sigma'(t) = \frac{-1}{(1 + e^{-t})^2} \cdot (-e^{-t}) = \frac{e^{-t}}{(1 + e^{-t})^2}$.
2. Equivalently $\sigma'(t) = \frac{1}{1 + e^{-t}} \cdot \frac{e^{-t}}{1 + e^{-t}}$; the first factor is $\sigma(t)$.
3. Multiplying the second factor by $e^t/e^t$ gives $\frac{1}{e^t + 1} = \frac{1}{1 + e^{t}} = \sigma(-t)$.
4. Hence $\sigma'(t) = \sigma(t)\,\sigma(-t)$. $\blacksquare$

The origin is in $H$ iff $w \cdot \mathbf{0} = b$, i.e. iff $b = 0$. Distance: $w \cdot p = 2$, $|w| = \sqrt{4 \cdot \frac14} = 1$, so $\text{dist}(p, H) = \frac{|2 - 0|}{1} = 2$.

($\Leftarrow$) $P = R = 1$ gives $F_1 = 2/2 = 1$. ($\Rightarrow$) $F_1 = 1$ means $1/P + 1/R = 2$; since $0 < P, R \le 1$ we have $1/P \ge 1$ and $1/R \ge 1$, so both equal $1$. XOR-like points: $(1,1)$ and $(2,2)$ in class 0, $(2,1)$ and $(1,2)$ in class 1. $F_1 = 1$ for both classes needs all points classified correctly, but class 0 sits on one diagonal and class 1 on the other, and no straight line separates them. (Proof by the midpoint argument in Linear Classifier > Properties.)

$M(x, w) = \sigma(w \bullet x)$ with $\sigma(z) = \frac{1}{1 + e^{-z}}$, and the binary cross-entropy $L(y, y') = -[y \ln y' + (1 - y)\ln(1 - y')]$.

def epoch(X: torch.Tensor, y: torch.Tensor, w: torch.Tensor) -> torch.Tensor:
    for i in range(X.shape[0]):
        xi = X[i]    # the i-th input vector, a 1D tensor
        yi = y[i]    # the i-th label, a single float
        factor = 0.1 * (yi - torch.sigmoid(torch.dot(xi, w)))
        w = w + factor * xi
    return w

For cross-entropy loss, the gradient step for one example is $w \leftarrow w + \eta\,(y_i - \sigma(w \cdot x_i))\,x_i$ with $\eta = 0.1$; the bracket is the error between label and predicted probability.

$\text{softmax}(x, 0) = \left(\frac{e^x}{e^x + e^0}, \frac{e^0}{e^x + e^0}\right) = \left(\frac{e^x}{e^x + 1}, \frac{1}{e^x + 1}\right)$. Multiplying numerator and denominator of the first component by $e^{-x}$ gives $\frac{1}{1 + e^{-x}} = \sigma(x)$; the second is $\frac{1}{1 + e^{x}} = \sigma(-x)$. So $\text{softmax}([x, 0]) = (\sigma(x), \sigma(-x))$, consistent with $\sigma(x) + \sigma(-x) = 1$.