Introduction
Training happens once (or occasionally); inference happens every time someone uses the model, possibly millions of times a day. For a popular service, inference is usually where most of the compute cost goes, and users notice every bit of delay.
Serving a model means running it behind an interface (like a web API) that other programs can call. Doing it well means balancing three things:
- latency: how long one request takes (e.g. under 100 ms for an interactive feature);
- throughput: how many requests per second the system handles;
- cost: how many GPUs or CPUs that requires.
Think of a coffee shop. Latency is how long one customer waits; throughput is customers served per hour; cost is the number of baristas and machines. Batching orders, preparing popular items in advance and using faster machines all help, as long as no one waits too long.