Introduction
In the last topic we met underfitting and overfitting. The bias–variance trade-off is the idea that ties them together and explains why they happen.
Picture an archer shooting arrows at a target, practising on different days:
- Bias is about aim: do the arrows land, on average, around the centre — or consistently off to one side?
- Variance is about consistency: are the arrows tightly grouped, or scattered all over the place from one day to the next?
A good archer has low bias (aims at the centre) and low variance (arrows close together). In ML, each "day" is a different training dataset, and each "arrow" is the model trained on it.
| Low variance | High variance | |
|---|---|---|
| Low bias | 🎯 accurate and consistent — ideal | right on average but scattered |
| High bias | consistently wrong in the same way | wrong and scattered — worst |