Introduction
Training a model means finding parameters that make a loss small. For most models there is no formula for the best parameters, so we search: start somewhere and repeatedly take a step downhill. Gradient descent is that idea, and almost every modern optimizer — SGD, momentum, Adam — is a refinement of it.