101 puffs

Chapter 1 · How machines learnPuff 2 of 23

How wrong is it?

Draw two lines through the café’s ten days of sales and both can look about right. To pick the better one, and to make a line better bit by bit, a machine needs to boil “how wrong” down to a single number. There is more than one way to do that, and the choice changes which line wins.

Guess first

One day a line misses by 10 coffees, another day by 2. If we square each miss, how much more does the big one count?

Play with it

Fig. 1 · Scoring a line Every miss, as a stick or as a square

Drag the two pink handles to move the line. Each dashed stick is one day’s miss. Press “Draw squares” to turn every miss into a square.

How it works

Tap a step to see it in the figure.

Check yourself

Why do machines usually square the misses?

Go deeper: the mathTwo scores, their slopes, and the mean versus the median

The two scores

For each of the n days, the miss is the real sales yi minus the line’s guess ŷi. The two common scores are:

MSE = 1n Σ (yi − ŷi)²

MAE = 1n Σ |yi − ŷi|

The bars |…| mean “size, ignoring the sign”. For your pink line right now: MSE 82.6, MAE 7.8. The “typical miss” from the last puff is √MSE, which turns squared coffees back into plain coffees.

Why squares are smooth

Picture how each score changes as one day’s miss e shrinks towards zero. The slope of e² is 2e: a miss of 10 pushes back five times harder than a miss of 2, and the push fades smoothly to nothing as the miss disappears. The slope of |e| is always +1 or −1: every miss pushes equally hard, and the push flips suddenly at zero, where the slope isn’t even defined.

dde e² = 2e     dde |e| = ±1 (+1 when the miss is positive, −1 when negative, undefined at exactly 0)

Learning, as the next puff shows, means following slopes downhill. Smooth slopes that shrink near the answer make that easy. Squared error also has a neat formula for the best line, least squares, and it is the statistically right choice when the day-to-day noise follows a bell curve.

Mean or median

The simplest model of all guesses the same sales every day. Under squared error the best single guess is the mean (the average); under absolute error it is the median (the middle value). For the café’s days: mean 41.7, median 40.5. Add the festival day, then drag it higher: the mean climbs with it, but the median stays put, because it only cares which side of the middle a day falls on, not how far away it is. The two best lines in the figure behave the same way.

Right now the best lines are: least squares ŷ = 2.87x − 21.9; least absolute ŷ = 2.75x − 19.3. The page finds the least-squares line with its formula and the least-absolute line by checking every line that passes through two of the days, since there is always a best line that passes through at least two.

What this page simplifies

Here the model guesses a number, so the miss is easy to measure. A chatbot guesses the next word, and “how far off” a word is needs a different score, called cross-entropy, which you will meet in puff 12. The job is the same: one number that says how bad the guess was.

Sources

  1. Google, Machine Learning Crash Course: Linear regression, Loss.
  2. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, chapter 5 (section 5.5.1 links squared error to bell-shaped noise).
  3. P. Bloomfield and W. L. Steiger, Least Absolute Deviations: Theory, Applications and Algorithms, Birkhäuser, 1983.
  4. The café numbers are made up for this course, and so is the festival day.