Introduction
How much "information" does an outcome carry? A rare event is surprising and tells you a lot; a near-certain one tells you almost nothing. Information theory turns this intuition into numbers. Its central quantity, entropy, measures the uncertainty of a distribution — and leads directly to the cross-entropy loss used to train almost every classifier and language model.