Introduction
When a human translates a long sentence, they don't memorize the whole sentence, close their eyes, and then recite the translation. They look back at the source sentence, focusing on the relevant words for each word they write. When writing the French word for "cat", they look at "cat"; when writing the verb, they look at the verb.
Attention gives neural networks the same ability. Instead of squeezing the whole input into one fixed vector, the model keeps all the encoder's outputs and, at each step, computes how much to focus on each one. It was introduced for translation in 2014 and turned out to be one of the most important ideas in deep learning: it's the core of the transformer (topic 4.4), and therefore of BERT, GPT and nearly every modern language, vision and multimodal model.