Introduction
Labelled data is expensive: someone has to look at every image or sentence and write down the answer. Unlabelled data is everywhere: billions of photos, videos, web pages and audio recordings. What if a network could learn from those, without any human labels?
Self-supervised learning does exactly that. The trick is to invent a task whose answers come from the data itself. Hide part of a sentence and predict the missing word. Rotate an image and predict the rotation. Show two crops of the same photo and learn that they belong together. To solve these tasks, the network has to learn a lot about language or images, and those learned features can then be transferred to real tasks (topic 5.1) with only a few labels.
Think of a child learning a language. Nobody labels each sentence for them; they hear millions of sentences, guess what comes next, and notice when they're wrong. Self-supervised learning is how today's largest models (GPT, BERT, CLIP, DINO) learned most of what they know.