Introduction
So far we've seen two families of RL methods:
- policy gradients (topic 1.2): learn the policy directly. They handle continuous actions and stochastic policies naturally, but are noisy and slow to learn;
- value-based methods (topic 1.4): learn Q-values and act greedily. They're more sample-efficient, but awkward for continuous actions.
Actor-critic methods combine them. An actor (the policy) decides what to do; a critic (a value function) judges how good things are, and its judgement reduces the noise in the actor's updates. PPO, a carefully stabilized actor-critic method, is one of the most widely used RL algorithms today, including in the fine-tuning of large language models with human feedback.
Think of an actor rehearsing with a director. The actor tries a scene; the director doesn't wait for the audience's reaction on opening night (the final return), but says right away "that line was better than usual" or "worse than usual". The actor improves much faster with this immediate, informed feedback.