101 puffs

Chapter 5 · From model to chatbotPuff 19 of 23

Reading the internet

Before a chatbot can chat, it reads. A lot. Its first lesson, called pretraining, is one simple game played trillions of times: here is some text, guess the next word. Nobody hands it a list of facts. So how does it end up knowing that Paris is in France?

Guess first

Nobody teaches the model facts during pretraining. So how does it end up “knowing” that Paris is in France?

Play with it

Fig. 1 · Pretraining a tiny model Read more, guess better
The model continues the text:

the sea is …

Each word is picked from its guesses for the next word (temperature 0.8, same dice rolls every time).

Slide to let the model read more of the Puff stories. Watch the sample turn from word salad into sentences, and its score on two stories it never reads fall.

The model above is a word-counting trigram from puff 12: it tallies which word followed each pair of words. Real chatbots compute their guesses with a neural network (the transformer from chapter 4) instead of a tally, but the game is the same. What changes is the amount of reading.

Fig. 2 · How much real models read Tokens of text seen in pretraining

Each step along this scale is a thousand times more text than the one before. Flip to an ordinary scale to see why that’s needed.

How it works

Tap a step to see it in the figure.

Check yourself

A model has only been pretrained. You type “What is the capital of France?” What might it do?

Go deeper: the mathThe pretraining score, smoothing and scaling laws

The one objective

Pretraining shrinks one number: the average surprise at each real next word. If the model gave the real word wt probability p, its surprise is −log p: zero when it was certain and right, large when it thought the word unlikely. Averaged over T words of text:

L = − 1T Σt log p(wt | the words before it)

This is the loss from chapter 1, the number that gradient descent pushes downhill. A friendlier version is the perplexity, eL: the number of words the model is, on average, torn between. Pure guessing among the 210 different words in the Puff stories would give a perplexity of 210.

perplexity = eL

Right now, on the two unseen stories, the page’s model scores L = ….

Why smoothing is needed

A pure tally gives probability 0 to any word it never saw in that spot, and −log 0 is infinite. One unseen word would ruin the whole score. So the page mixes four guesses, a method called interpolation:

p(w | u v) = 0.4 · count(u v w)count(u v) + 0.3 · count(v w)count(v) + 0.2 · count(w)N + 0.1 · 1210 What followed the last two words, what followed the last word, how common the word is, and a small share for any word at all. If the model has never seen a context, its share goes to the others.

For example, …

The weights 0.4, 0.3, 0.2 and 0.1 are hand-picked for this page. The sample text above uses the plain tally (falling back to shorter contexts when needed), so it only ever says words it has actually read.

Scaling laws

Real models show the same shape as Fig. 1, only smoother: the loss falls steadily as the model, the text and the computing power grow. Kaplan and colleagues (2020) found this follows simple power laws: each tenfold increase buys a predictable drop in loss. Hoffmann and colleagues (2022) then showed that, for a fixed computing budget, many models had too many knobs and too little reading. Their model Chinchilla, with 70 billion knobs trained on 1.4 trillion tokens, beat much larger models that had read less.

What this page simplifies

The page’s model is a tally of word triples, trained on 612 words and tested on 104 more; its “tokens” are whole words and punctuation marks. Real models chop text into word pieces (puff 10), read trillions of them, and set billions of knobs by gradient descent rather than counting. For a tally, counting already gives the best possible fit; for a neural network there is no such shortcut, so it nudges, checks and repeats.

Sources

  1. T. B. Brown et al., Language Models are Few-Shot Learners, 2020. GPT-3 was trained on 300 billion tokens.
  2. H. Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. Trained on 2 trillion tokens.
  3. Llama Team, AI @ Meta, The Llama 3 Herd of Models, 2024. About 15 trillion tokens (15.6 trillion for the largest model).
  4. J. Kaplan et al., Scaling Laws for Neural Language Models, 2020; J. Hoffmann et al., Training Compute-Optimal Large Language Models, 2022.
  5. The training text is the Puff stories, written for this site. The last two stories are held back for testing.