Introduction
Many problems come with rules: probabilities must sum to 1, a budget can't be exceeded, a weight vector must have bounded norm, a classifier's margin must be at least 1. Constrained optimization finds the best point that satisfies the rules. Lagrange multipliers turn a constrained problem into an unconstrained one with extra variables — the technique behind PCA's derivation, maximum-entropy models, and support vector machines.