101 puffs

Chapter 1 · How machines learnPuff 4 of 23

Big steps vs. small steps

Rolling downhill only tells the machine which way to go. It still has to decide how far to move each time. Tiny steps are safe but slow. Big steps are fast, until one carries it right over the bottom of the valley and up the far side. How big should a step be?

Guess first

What happens if every step is far too big?

Play with it

Fig. 1 · The learning rate Thirty steps down a toy valley

Pick a step size with the slider, or try the four presets. The ball always starts at the same place and takes up to 30 steps.

How it works

Tap a step to see it in the figure.

Check yourself

Training was going fine, then the error suddenly shot up to huge numbers. What do you try first?

Go deeper: the mathOne multiplication explains all four behaviours

The toy valley

The figure uses the simplest possible valley, a bowl with steepness a and its bottom at w*:

L(w) = a (w − w*)² Here a = 1, w* = 3, and the ball starts at w = 0.

Its slope is 2a(w − w*), so one step of gradient descent with learning rate η works out as:

wnew − w* = (1 − 2aη) × (w − w*)

Every step multiplies the distance to the bottom by the same number, 1 − 2aη. With your step size: 1 − 2 × 1 × 0.35 = 0.3.

Four behaviours from one number

  • Between 0 and 1: the ball stays on the same side and closes in. Close to 1 means crawling.
  • Exactly 0, when η = 1/(2a): it lands on the bottom in a single step.
  • Between −1 and 0: it overshoots every time, landing on alternate sides, but closer each time.
  • Below −1, when η > 1/a: each landing is further away than the last, and the distance grows without limit.

So the ball settles only when 0 < η < 1/a. For this bowl that edge is η = 1. After k steps the distance from the bottom is 3 × |1 − 2aη|k; for your step size that is 3 × 0.3⁵ = 0.0073 after 5 steps.

Steeper valleys need smaller steps

Double the steepness a and the safe limit halves. A real network has a separate steepness in every direction of its huge landscape, and one step size has to suit them all: the steepest direction sets the speed limit, so the gentlest directions crawl. (In general, the limit is 2 divided by the largest curvature, the second derivative; for this bowl that is 2/(2a) = 1/a again.) This is also why puff 1 rescaled the temperatures: it made the valley similarly steep for both knobs.

What real training does

Real training rarely uses one fixed step size. A schedule changes it over time: a warm-up that starts small and grows, then a decay that shrinks it towards the end so the knobs can settle. Adaptive optimisers such as Adam go further and give every knob its own step size, scaled by how large that knob’s recent slopes have been. The bowl here is a toy, picked so the maths comes out exact; real error landscapes are bumpier, but near a low point many look roughly like a bowl.

Sources

  1. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, chapter 8 (learning rates in 8.3, adaptive methods in 8.5).
  2. D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization, 2014.
  3. P. Goyal and others, Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 2017 (gradual warm-up).
  4. Google, Machine Learning Crash Course: Linear regression, Hyperparameters.
  5. The valley is a toy bowl made up for this page, so that every number follows exactly from the formula.