Introduction
A bounding box is a rough answer: it tells you an object is somewhere inside a rectangle. Many tasks need exact shapes. A self-driving car needs to know precisely which pixels are road. A doctor needs the exact outline of a tumour. A photo app needs to cut a person out of the background.
Segmentation gives a label to every pixel:
- semantic segmentation: each pixel gets a class (road, car, person, sky). Two people next to each other are both just "person" pixels;
- instance segmentation: each pixel gets a class and an object identity (person 1, person 2, car 1…);
- panoptic segmentation combines both: every pixel gets a class, and countable objects also get instance identities.
Think of a colouring book. Semantic segmentation colours all people red and all cars blue. Instance segmentation gives each person their own shade.