Introduction
Imagine a game of "telephone" with 50 people in a line. The last person hears the message and whispers a correction back down the line. If each person whispers a little more quietly than they heard it, the correction fades to silence long before it reaches the front. If each person shouts a little louder, it turns into a deafening, garbled roar.
Backpropagation is exactly this game. The error signal (the gradient) starts at the output and is passed backwards, layer by layer, to the first layer. At every layer it gets multiplied by something. Multiply by numbers smaller than 1 many times and the signal vanishes. Multiply by numbers larger than 1 many times and it explodes.
For about two decades this was the main reason people could not train networks with more than a few layers. Most of this chapter is about the fixes that made deep learning possible.