Introduction
By default, PyTorch stores every number as a 32-bit float (FP32). That's very precise, more precise than neural networks usually need. Modern GPUs have special hardware (Tensor Cores) that multiplies 16-bit numbers many times faster than 32-bit ones, and 16-bit numbers take half the memory.
Mixed precision training does most of the heavy computation (matrix multiplications, convolutions) in 16-bit, while keeping the few precision-sensitive parts (weight updates, loss, some reductions) in 32-bit. You typically get 1.5–3× faster training and much lower memory use, with essentially the same accuracy.
Think of an architect's office. Rough sketches are drawn quickly in pencil (16-bit), which is perfectly good for most of the work. Only the final official measurements are written in ink with a precise ruler (32-bit).