Introduction
Running diffusion directly on pixels is expensive: a 512 × 512 colour image has about 786,000 values, and the network must process all of them at every one of many steps. And an unconditional model generates random images, while what people want is to type "an astronaut riding a horse in watercolour" and get exactly that.
Three ideas solved both problems and produced Stable Diffusion (Rombach et al., 2022) and similar text-to-image systems:
- latent diffusion: compress images with an autoencoder and run diffusion in the much smaller latent space;
- conditioning: feed a text description into the denoising network through cross-attention;
- classifier-free guidance: make the output follow the prompt more strongly.
Think of an architect. Instead of building a full-size house over and over to try ideas, they work on a compact blueprint (the latent space) and only build the real house (decode to pixels) at the end. And they follow the client's written brief (the text prompt) at every step of the design.