Theory of Intelligence
MATH FOR AI•Optimization•Gradient Descent•1.2 Step Size

Introduction

What happens when the learning rate is too small or too large, and learning-rate schedules: step, cosine and warm-up.

timer1 min read•format_list_bulletedConcept 1 of 3•forumLesson Discussion

Introduction

The learning rate η\eta is the single most important hyperparameter in deep learning. Too small and training crawls; too large and it diverges. In practice the best results come from changing the learning rate during training with a schedule.

Discuss theoretical concepts, mathematical formulas, and architectural foundations.

Supports inline code with `backticks`
No comments yet. Be the first to start the discussion!
arrow_backPrevious Lesson
1.1 Gradient Descent