ConceptDBISAI-written · not yet proofread

Lagrange Multipliers

Also Lagrangian Optimization · Lagrange Multiplier · Lagrangian

The method of Lagrange multipliers finds extrema of f(x) under an equality constraint g(x) = 0 by setting all partial derivatives of L(x, λ) = f(x) + λ g(x) to zero.

Properties#

Candidates can be maxima, minima or saddle points. The exams either allow assuming “no saddle points” (WS 22/23) or ask to check the second derivative (WS 16/17 hint).
Typical use in the course: probability distributions, with constraint $\sum_i p_i - 1 = 0$ (maximum entropy, Maximum Likelihood Estimation, entropy-optimal code lengths).

Example#

Sources#

WS 13/14 Task 5.3 (not published) (reminder on how to set up $L$ ), WS 16/17 Task 6.3 (not published), WS 22/23 Task 1.3 (not published)
Not covered in the WS 25/26 lectures.

Connections

Mentioned in inline links2

Maximum Likelihood Estimation
With a Lagrange multiplier: $\partial/\partial p_i \,[\sum_j c_j \ln p_j + \lambda(1 - \sum_j p_j)] = c_i/p_i - \lambda = 0$ , so $p_i \propto c_i$ and §
Encodings and Compression
Lagrange Multipliers — proof tool: minimising $\sum_i p_i L_i$ under the Kraft constraint gives $L_i = \log_2(1/p_i)$ §
Prove the optimum: minimise $\sum_i p_i L_i$ subject to $\sum_i 2^{-L_i} = 1$ with Lagrange Multipliers, which yields $p_i = 2^{-L_i}$ . §

Search in these mentions →