Introduction
A new recommendation model has better offline accuracy than the old one. Does that mean users will watch more, buy more or come back more often? Not necessarily:
- offline metrics measure how well the model matches past behaviour, not how users react to new recommendations;
- a model can be "more accurate" at predicting clicks yet recommend less diverse content that users tire of;
- latency, UI changes and feedback loops only show up live.
An A/B test is the way to find out. Randomly split users into two groups: A (control) keeps the current model, B (treatment) gets the new one. Compare a business metric. Because the split is random, any significant difference is caused by the model change (Causal Inference, topic 2.4).