Introduction
A modality is a kind of data: text, images, audio, video, 3-D shapes, sensor signals. Multimodal generative models take one or more modalities as input and generate another: text → image, text → speech, image → text, speech → speech, text → video.
The good news is that you already know the building blocks. Almost every multimodal generator combines a few ideas from this course:
- encoders that turn each modality into vectors or tokens (CNNs, ViTs, audio encoders, text transformers);
- a shared space or cross-attention that connects modalities (CLIP, Computer Vision tab topic 4.3; Sequences tab topic 4.4);
- a generator: an autoregressive transformer (topic 4.1) or a diffusion model (Chapter 03), often working in a compressed latent space.
Think of a film studio. The script (text) goes to a storyboard artist (image), a composer (music), voice actors (speech) and an animation team (video). Each specialist speaks a different "language", but they all work from a shared understanding of the story. Multimodal models build that shared understanding inside one system.