Introduction
A simple RNN rewrites its whole memory at every step (topic 1.4), so old information fades quickly. The LSTM cell (long short-term memory, Hochreiter and Schmidhuber, 1997) fixes this by giving the network a separate, protected memory, the cell state, and gates that decide what to forget, what to write and what to read at each step.
Think of a notebook carried through a long meeting. At each moment you decide:
- which old notes are no longer relevant, and cross them out (forget gate);
- which new information is worth writing down (input gate);
- which notes to read out when someone asks you a question right now (output gate).
Notes you don't cross out stay in the notebook unchanged, however long the meeting lasts. That's how an LSTM can remember things for hundreds of steps.