Introduction
Evaluating a classifier is easy: compare its predictions with the true labels. Evaluating a generative model is much harder. There's no single "right answer" for "draw a cat" or "write a poem about rain". We want samples that are:
- high quality: realistic, sharp, fluent, coherent;
- diverse: covering the full variety of the data, not the same few outputs (topic 2.2);
- faithful to the condition: matching the prompt or class;
- novel: not just copies of training examples;
- safe: not harmful, biased or deceptive.
No single number captures all of this. Think of judging a cooking competition: you taste each dish (quality), check that the chef can cook different cuisines (diversity), that they made what was ordered (faithfulness), and that they didn't just buy it from a shop (novelty). Practitioners combine automatic metrics with human judgement.