Introduction
Left alone, a decision tree will keep asking questions until every leaf is pure — until it classifies every training example correctly. On real, noisy data that means building strange, tiny branches for individual unusual examples. The result is a tree that has memorised the training set and generalises badly: classic overfitting.
Trees are called nonparametric models: the number of parameters isn't fixed in advance — the tree grows as big as the data allow. That flexibility is great, but it must be restrained. Linear models are parametric: their number of parameters is fixed (one weight per feature), so they tend to overfit less but may underfit.
Restraining a tree is called regularization, and there are two families of methods: limit growth while building the tree (pre-pruning), or build a big tree and cut it back afterwards (post-pruning).