Introduction
Entropy measures the uncertainty of one distribution. To compare two distributions — the true one and the model's — we use cross-entropy and KL divergence. Cross-entropy is the loss function for classification and language modelling; KL divergence appears in VAEs, knowledge distillation and the fine-tuning of chatbots with human feedback.