Introduction
Models trained step by step — with gradient descent, boosting or neural networks — get better at the training data with every step. But at some point they stop learning general patterns and start memorising the training set. If you keep going, you're actually making the model worse for new data.
Early stopping is the obvious fix: watch the validation error during training, and stop at the point where it was lowest. Geoffrey Hinton famously called it a "beautiful free lunch" — it's simple, cheap and very effective.
Think of baking bread: the bread gets better in the oven, but leave it too long and it burns. You don't fix that with a different recipe — you just take it out at the right moment.