Theory of Intelligence
DEEP LEARNING•Computer Vision•Vision Transformers And Modern Vision•4.1 Vision Transformers (ViT)

Introduction

Treating an image as a sequence of patches and processing it with a transformer: how ViT works, how it compares with CNNs, and how DeiT and Swin made it practical with less data.

timer1 min read•format_list_bulletedConcept 1 of 4•forumLesson Discussion

Introduction

Transformers were invented for text: they read a sentence as a sequence of tokens and let every token "pay attention" to every other token (Sequences tab, topic 4.2). In 2020, a Google team asked a bold question: what if we treat an image as a sequence too, and feed it to an almost unchanged transformer?

The answer was the Vision Transformer (ViT) (Dosovitskiy et al., "An Image is Worth 16x16 Words"). Trained on enough data, it matched or beat the best CNNs. Today, transformer-based vision models are behind many state-of-the-art systems, including the image side of multimodal models like CLIP (topic 4.3).

Think of describing a photo by cutting it into a grid of square tiles, like a jigsaw puzzle, and laying the tiles out in a row. Each tile is a "word". The transformer reads all the words together, and works out how the tiles relate to each other to understand the whole picture.

Discuss theoretical concepts, mathematical formulas, and architectural foundations.

Supports inline code with `backticks`
No comments yet. Be the first to start the discussion!
Next Lessonarrow_forward
4.2 Self-Supervised Vision