Introduction
A training loss is an average over the whole dataset — maybe millions of examples. Computing its exact gradient for every step would be far too slow. Stochastic gradient descent (SGD) estimates the gradient from a small random mini-batch instead. Each step is noisy but cheap, and many cheap noisy steps beat a few expensive exact ones. Every deep network is trained this way.