Introduction
Deep networks have millions of parameters: more than enough to memorize the training set, noise and all. That's overfitting: great training accuracy, disappointing validation accuracy. Regularization is any technique that makes it harder for the network to memorize and nudges it towards simpler, more general solutions.
The oldest regularizer is to keep the weights small. Large weights let a network build very sharp, wiggly functions that pass exactly through every training point. Small weights produce smoother functions that generalize better.
Think of a gardener trimming a hedge. Without trimming, branches grow wildly in every direction to grab every bit of light. A regular light trim (weight decay) keeps the hedge compact and tidy, without stopping it from growing where it really needs to.