Theory of Intelligence
DEEP LEARNING•Training Deep Networks•Unstable Gradients•1.4 Batch & Layer Normalization

Introduction

Normalization layers that re-centre and re-scale activations inside the network: how batch norm and layer norm work, how they differ, and where to put them.

Introduction

In the ML track, you learned to standardize input features (mean 0, standard deviation 1) before training. It makes optimization much easier. But in a deep network, only the first layer sees those nicely scaled inputs. Every later layer gets the outputs of the layer before it, and those can drift to any scale as the weights change during training.

Normalization layers apply the same idea inside the network: before (or after) a layer's activation, they rescale the values so they're centred near 0 with a spread near 1. Then they let the network learn whatever final scale and shift it actually wants.

Think of a relay race where each runner has to hand a baton to the next. If some runners hand it over far too high and others far too low, the next runner wastes effort adjusting. A normalization layer is a coach at each hand-off point who makes sure the baton is always passed at a standard height.

Discuss theoretical concepts, mathematical formulas, and architectural foundations.

Supports inline code with `backticks`
No comments yet. Be the first to start the discussion!