Introduction
Offline metrics (accuracy, AUC) don't guarantee that a new recommendation or ranking model will help real users. Companies decide by running A/B tests: show the old model (A) to some users and the new one (B) to others, then compare a business metric such as click-through or retention. It is hypothesis testing applied to live systems, with its own design choices and traps.