Introduction
The regularizers so far act on the weights (weight decay, max-norm), the neurons (dropout) or the data (augmentation). This topic adds two that act in less obvious places:
- label smoothing changes the targets the network is trained on, so it stops chasing 100% certainty;
- stochastic depth randomly removes whole layers (residual blocks) during training, making a deep network behave like an ensemble of shallower ones.
Both are cheap, simple to add, and part of the standard recipes for training modern image classifiers and transformers.