Introduction
One learning rate for millions of parameters is a compromise: some parameters see large, frequent gradients, others small or rare ones. Adaptive optimizers give each parameter its own effective step size, based on the history of its gradients. AdaGrad introduced the idea; RMSProp made it work for long deep-learning runs; Adam (next topic) combined it with momentum.