Introduction
Plain gradient descent zig-zags across narrow valleys and crawls along their floor. Momentum fixes this by letting the optimizer remember its recent direction, like a heavy ball rolling downhill: oscillations across the valley cancel out, while consistent motion along it builds up speed. Almost all deep-learning optimizers use some form of momentum.