Introduction
Diagonal matrices are the easiest matrices there are: they just scale each axis. Eigendecomposition shows that many matrices are secretly diagonal — if you look at them in the basis of their eigenvectors. For symmetric matrices (like covariances) this always works and the basis is even orthonormal. This is the mathematical core of PCA.