Introduction
The previous topics prevent unstable gradients by setting things up well: good initial weights, good activations, normalization. This topic adds two more tools:
- gradient clipping is a safety net: if a gradient gets too big anyway, shrink it before taking the step;
- residual connections change the architecture so gradients have a direct shortcut from the output to every layer.
Think of driving down a steep mountain road. Gradient clipping is a speed limiter: whatever happens, the car never goes faster than a safe maximum. Residual connections are like a tunnel straight through the mountain: the signal doesn't have to follow every hairpin bend.