Introduction
Not every model should run in the cloud. Your phone unlocks with your face, transcribes voice messages, blurs your video-call background and suggests the next word as you type, often without sending anything to a server. A car's driver-assistance system can't wait for a network round trip. A sensor in a field may have no internet at all.
Running models on the device brings real benefits:
- privacy: data (photos, voice, health readings) stays with the user;
- latency: no network round trip, so responses are instant;
- offline operation;
- cost: no server compute bill for each prediction.
The challenge is that devices have tight budgets: limited memory (sometimes kilobytes on a microcontroller), modest processors, battery life and heat limits. Think of packing for a long hike: you can't bring the whole kitchen, so you choose light, multi-purpose equipment and pack it efficiently. On-device deep learning is about choosing light models and packing them efficiently.