Theory of Intelligence
DEEP LEARNING•Computer Vision•Cnn Architectures•2.3 ResNet & Skip Connections

Introduction

Why simply stacking more layers made networks worse, how residual blocks fixed it, and how ResNet made networks with 50, 100 and even 1,000 layers trainable.

timer1 min read•format_list_bulletedConcept 1 of 4•forumLesson Discussion

Introduction

By 2015, it seemed obvious that deeper is better: VGG and GoogLeNet beat AlexNet largely by adding layers. So why not 50 layers? 100? Researchers tried, and found something surprising: beyond about 20 layers, plain networks got worse, even on the training set.

ResNet (Kaiming He et al., Microsoft Research, 2015) solved this with one small change: skip connections that add a block's input to its output. It won ImageNet 2015 with a 152-layer network and a top-5 error of about 3.6%, better than the estimated human level. Residual connections are now in nearly every deep architecture, including transformers.

Think of editing a document. Rewriting the whole thing from scratch at every revision is risky; one bad revision can ruin it. Instead, each editor makes small corrections to the current version (tracked changes). Even if one editor adds nothing useful, the document is no worse. A residual block is an editor that makes tracked changes.

Discuss theoretical concepts, mathematical formulas, and architectural foundations.

Supports inline code with `backticks`
No comments yet. Be the first to start the discussion!
arrow_backPrevious Lesson
2.2 VGG & GoogLeNet
Next Lessonarrow_forward
2.4 Efficient & Modern CNNs