Theory of Intelligence
DEEP LEARNING•Sequences And Transformers•Recurrent Neural Networks•1.2 Training RNNs

Introduction

How recurrent networks learn: backpropagation through time on the unrolled network, and truncated BPTT for long sequences, with a complete PyTorch training example.

Introduction

An RNN unrolled over TT time steps looks like a deep network with TT layers that all share the same weights (topic 1.1). Training it uses the same idea as for any network: compute a loss, then use backpropagation to find how each weight should change. The twist is that each weight is used at every time step, so its gradient must collect contributions from all of them.

Think of a relay race where the same runner runs every leg. To give them feedback, the coach looks at each leg separately ("too slow at the start of leg 3", "good at the end of leg 7") and then adds up all those notes into one overall piece of advice. That's backpropagation through time.

Discuss theoretical concepts, mathematical formulas, and architectural foundations.

Supports inline code with `backticks`
No comments yet. Be the first to start the discussion!