Memorising vs. understanding

The model

The curves are polynomials: a sum of powers of the temperature x, each with its own knob. The wiggliness is the highest power, the degree d, and the curve has d + 1 knobs:

ŷ = w₀ + w₁x + w₂x² + … + w_dx^d

For each degree the page finds the knobs with the lowest mean squared error on the 13 training days, the same least-squares score as before.

Two kinds of error

The “typical miss” is the square root of the mean squared error, so it is measured in coffees. The page computes it twice: on the training days, which the knobs were fitted to, and on the 20 held-out days, which they never saw (often called the test set):

typical miss = √( 1n Σ (ŷ_i − y_i)² )

Right now: degree 1: training miss 8.1, held-out miss 7.8. The lowest held-out miss comes at degree 3. Training error never goes up as knobs are added; held-out error falls, then rises again. That U shape is the signature of overfitting.

Too rigid or too flexible

A model’s capacity is how many different shapes it can take, which grows with its number of knobs. Too little, and it is wrong in the same way whatever data it sees; statisticians call that bias. Too much, and it bends to whatever random luck the training days happened to have, so a fresh sample of days would give a completely different curve; that is variance. The best model balances the two.

Fighting overfitting

• More data. The more days there are, the harder it is to thread them all, and the noise averages out.
• A simpler model. Fewer knobs, as with the slider.
• Regularisation. Keep all the knobs, but add a penalty for large knob values to the score, such as λ Σ w_j². Wild wiggles need large knobs, so they get discouraged.
• Early stopping. Watch the held-out error during training and stop when it starts to rise.

In practice there are three piles of data: training data to fit the knobs, a validation set to choose settings like the degree, and a test set looked at only once, at the very end. Choosing the degree by peeking at the test set, as this page invites you to, would make its score look better than it really is.

What this page simplifies

Huge neural networks complicate the simple U shape: with far more knobs than data points they can still do well on new data, a puzzle studied under the name “double descent”. But judging a model on data it has never seen is standard practice for models of every size, chatbots included.

For the curious: the page rescales temperature to run from −1 to 1 and builds the curves from Legendre polynomials, solved with a QR decomposition. That gives exactly the same best curves as plain powers of x, but stays accurate at degree 12, where the plain version runs into rounding errors.