Introduction
Simple RNNs work well for short patterns, like the weekly cycle in the last topic. But ask one to remember something from 100 steps ago, like the subject of a long sentence or a spike in a sensor reading two days earlier, and it usually fails.
There are two related problems:
- unstable gradients: when training through many time steps, gradients vanish or explode, so the network can't learn long-range patterns;
- short-term memory: even in the forward pass, the hidden state is overwritten a little at every step, so information from early steps fades away.
Think of a game of "telephone" again (Training tab, topic 1.1), but this time with 100 people and a message that changes slightly at each person. By the end, almost nothing of the original message remains. Simple RNNs are like that. This topic covers the partial fixes, and why they led to the gated cells of Chapter 02.