Definition
Given observations x 1 , … , x N x_1, \dots, x_N x 1 , … , x N θ \theta θ L ( θ ) = Pr ( x 1 , … , x N ∣ θ ) L(\theta) = \Pr(x_1, \dots, x_N \mid \theta) L ( θ ) = Pr ( x 1 , … , x N ∣ θ ) likelihood ), the maximum likelihood estimate is θ ^ = arg max θ L ( θ ) \hat\theta = \arg\max_\theta L(\theta) θ ^ = arg max θ L ( θ ) ln L \ln L ln L
For a die with outcomes 1 , … , d 1, \dots, d 1 , … , d p 1 , … , p d p_1, \dots, p_d p 1 , … , p d ∑ i p i = 1 \sum_i p_i = 1 ∑ i p i = 1 c i c_i c i i i i
L ( p ) = ∏ i p i c i , ln L = ∑ i c i ln p i . L(p) = \prod_i p_i^{c_i}, \qquad \ln L = \sum_i c_i \ln p_i. L ( p ) = i ∏ p i c i , ln L = i ∑ c i ln p i . With a Lagrange multiplier : ∂ / ∂ p i [ ∑ j c j ln p j + λ ( 1 − ∑ j p j ) ] = c i / p i − λ = 0 \partial/\partial p_i \,[\sum_j c_j \ln p_j + \lambda(1 - \sum_j p_j)] = c_i/p_i - \lambda = 0 ∂ / ∂ p i [ ∑ j c j ln p j + λ ( 1 − ∑ j p j )] = c i / p i − λ = 0 p i ∝ c i p_i \propto c_i p i ∝ c i
p ^ i = c i N . \hat p_i = \frac{c_i}{N}. p ^ i = N c i . The MLE is the relative frequency. This is also how the n-gram models of DBIS_VL12 are “learned” by counting.
Example# Example WS 13/14 Task 5.3
A four-sided die shows 2 , 4 , 4 , 4 2, 4, 4, 4 2 , 4 , 4 , 4 p 2 ⋅ p 4 3 p_2 \cdot p_4^3 p 2 ⋅ p 4 3 c 1 = c 3 = 0 c_1 = c_3 = 0 c 1 = c 3 = 0 ln p 2 + 3 ln p 4 + λ ( 1 − p 2 − p 4 ) \ln p_2 + 3 \ln p_4 + \lambda(1 - p_2 - p_4) ln p 2 + 3 ln p 4 + λ ( 1 − p 2 − p 4 ) p 1 , p 3 p_1, p_3 p 1 , p 3 p 2 , p 4 p_2, p_4 p 2 , p 4 0 0 0 1 / p 2 = λ 1/p_2 = \lambda 1/ p 2 = λ 3 / p 4 = λ 3/p_4 = \lambda 3/ p 4 = λ p 4 = 3 p 2 p_4 = 3 p_2 p 4 = 3 p 2 p 2 = 1 / 4 p_2 = 1/4 p 2 = 1/4 p 4 = 3 / 4 p_4 = 3/4 p 4 = 3/4 p 1 = p 3 = 0 p_1 = p_3 = 0 p 1 = p 3 = 0
Sources# WS 13/14 Task 5.3 (not published) Not covered in the WS 25/26 lectures.