Chapter 3 · Words as numbersPuff 12 of 23
The one job: predict the next word
Watch a chatbot answer and the words appear one at a time, like someone typing. That isn’t a special effect. Each word really is chosen one at a time, by a model doing a single small job over and over. What is that job?
Guess first
When a chatbot writes a paragraph, what is it doing at each moment?
Play with it
The text so far words the model looks at
puff was
Its guesses for the next word tap one to add it
This tiny model counted which words follow which in twelve short stories about Puff. Tap a word to add it, or press “Pick the top word” to let the model choose. Keep going and it writes a story.
How it works
Tap a step to see it in the figure.
Check yourself
Where do this tiny model’s probabilities come from?
Go deeper: the mathCounting, chaining and the loss a model is trained on
The probability of the next word
A language model gives a probability for each possible next word w, given the words so far. A trigram model, like this one, only looks at the last two words, and gets the probability by counting:
P(w | a b) = count(a b w)count(a b)
Right now, for the top word:
…
When the last two words never appeared together, this model uses only the last word, and if that is new too, how common each word is overall. Real word-counting models smooth their counts in cleverer ways, so that unseen combinations still get a small chance.
A whole sentence, word by word
The probability of a whole piece of text is the product of the next-word probabilities along the way:
P(w1 … wn) = P(w1) × P(w2 | w1) × P(w3 | w1 w2) × …
The words you added so far had a combined probability of ….
How a model is scored: the loss
To train a model, show it real text, look at the probability p it gave to the word that actually came next, and take
loss = −ln pln is the natural logarithm, a button on most calculators
A word it was sure of (p = 1) costs nothing; a word it thought unlikely costs a lot. Averaged over lots of text, this is called the cross-entropy loss. Pretend your picks were the real text: …. A chatbot is trained by nudging its knobs to make this number smaller, over an enormous amount of text.
An old idea
In 1948 Claude Shannon wrote out text by choosing each word according to which words had followed the previous one in a book: a word-pair model by hand. In 2003 Yoshua Bengio and colleagues replaced the counts with a neural network that gives every word a vector (last puff), so similar words can share what they have learned. Today’s chatbots are the far bigger descendants of that network, but they still do this one job.
Sources
- C. E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, 1948.
- Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, A Neural Probabilistic Language Model, Journal of Machine Learning Research 3, 2003.
- D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd edition draft, chapter “N-gram Language Models”.
- The training text is twelve short Puff stories written for this site; every count and probability on this page is worked out from them.