Introduction
Self-attention compares every token with every other token (topic 4.2). For a 512-token input that's about 260,000 comparisons per head per layer, which is fine. For a 100,000-token document (a book, a codebase, hours of transcripts), it's 10 billion. Doubling the length quadruples the cost.
Meanwhile, users want models to read entire reports, long conversations and large code repositories at once. This topic covers how researchers and engineers made long contexts feasible:
- understanding the cost of attention;
- approximations that compute only part of the attention (sparse, linear);
- exact but faster implementations (FlashAttention) and the KV cache for fast generation.
Think of a meeting where everyone must talk one-on-one with everyone else. With 10 people, that's 45 conversations. With 1,000 people, it's about 500,000. You either limit who talks to whom (sparse attention), summarize groups (linear attention), or make each conversation extremely efficient (FlashAttention).