Introduction
Transformers are brilliant, but their attention cost grows with the square of the sequence length (Sequences tab, topic 5.3). For a million-token DNA sequence, hours of audio, or a very long document, that becomes impractical. RNNs (Sequences tab, Chapter 01) scale linearly and need only a fixed-size memory, but they're slow to train (step by step) and forget long-range information.
State space models (SSMs) aim for the best of both:
- like an RNN, they keep a fixed-size state that's updated at each step, so generation costs the same per token no matter how long the context is;
- unlike a classic RNN, their update is linear in a special structured way, so the whole sequence can be computed in parallel during training (like a long convolution);
- with careful design, they remember information over very long ranges.
Think of a ship's navigator using dead reckoning. They keep a compact record of the ship's state (position, speed, heading) and update it with each new measurement, without re-reading the entire logbook. An SSM keeps a compact, well-designed state that summarizes the whole history.