Introduction
Adam combines the two best ideas so far: momentum (a moving average of gradients, for direction) and RMSProp (a moving average of squared gradients, for per-parameter scale). It works well out of the box on a huge range of problems. Its variant AdamW, which fixes how weight decay interacts with adaptivity, is the standard optimizer for training Transformers and large language models.