Introduction
A single validation set has two weaknesses:
- It wastes data. The examples in it are never used for training.
- It is noisy. If the validation set happens to contain unusually easy or hard examples, your score is misleading. A different random split could give a noticeably different score.
Cross-validation solves both. Instead of one split, it makes several splits, so every example is used for training and for validation — just never at the same time. You then average the scores. It's like taking several practice exams made from different question banks, rather than judging yourself on one.