Introduction
Deriving every gradient entry by entry is slow. A small table of identities covers most of what appears in ML: linear models, quadratic losses, Gaussian likelihoods. Learn these and you can derive the normal equations, the Gaussian MLE, and many loss gradients in a few lines.
Convention below: gradients of a scalar with respect to a column vector are written as columns (same shape as ), the convention most ML code uses.