Introduction
A deep network can have billions of parameters, and training needs the gradient of the loss with respect to all of them, at every step. Writing these gradients by hand is impossible, and computing them naively would be far too slow. Backpropagation computes all of them in roughly the cost of one extra forward pass, by applying the chain rule backwards and reusing intermediate results. Automatic differentiation generalises this to any program.