Chapter 3 · Words as numbersPuff 10 of 23
Chopping text into tokens
You type “Puff saw an unbelievable thundercat” into a chatbot. It never sees those letters, and it doesn’t see your words either. Before anything else, a program chops the text into pieces called tokens and swaps each piece for a number. So where does it chop, and who decided?
Guess first
How does a chatbot see a long word like “unbelievable”?
Play with it
The tokens, with their ID numbers ▁ = a space
Latest merges
This tokeniser taught itself from the Puff stories a moment ago. Drag “Merges learned” down to 0 to see where it started, then back up to watch it glue letters into chunks. Tap any token to see where it came from.
How it works
Tap a step to see it in the figure.
Check yourself
Why not simply give every word its own token?
Go deeper: the algorithmByte-pair encoding step by step, and real tokenisers
Learning the merges
- Split the text into chunks. A chunk is a word together with the space in front of it, a run of punctuation, or a number. Merges never cross from one chunk to the next. This page uses GPT-2’s own splitting rule.
- Write each chunk as bytes. Plain English letters are one byte each; “a” is byte 97.
- Count every neighbouring pair, weighted by how often each chunk appears. “▁the” appears … times in the stories, so its pairs “▁”+“t”, “t”+“h” and “h”+“e” each get that many added.
- Merge the most common pair into one new token, with the next free ID number:
new token = the pair (a, b) with the biggest count(a, b) Its ID is the next free number: 256 for the first merge, 257 for the second, and so on.
- Repeat until the vocabulary is as big as you want. This page keeps going until no pair appears twice, which takes 278 merges.
To tokenise new text, the tokeniser replays its merges in the order it learned them. Right now it is using the first 278 merges. The most recent one is ….
Why bother?
Fewer, longer tokens mean less work for the model. With the current merges, the stories’ own 3,174 characters become … tokens, about … characters per token. Your text goes from ….
Real tokenisers
Byte-pair encoding began in 1994 as a trick for compressing files: replace the commonest pair of bytes with a spare byte, and repeat. In 2016 it was adapted to chop words into pieces for translation models, so they could handle rare words. GPT-2, a 2019 language model from OpenAI, used byte-level BPE with a vocabulary of 50,257 tokens: the 256 bytes, 50,000 merges and one special “end of text” token.
The algorithm on this page is the same; the difference is the training text. Real tokenisers learn their merges from enormous amounts of text, so they know far more chunks than ours, which learned from about 3,000 characters of stories. For ordinary English text, OpenAI’s rule of thumb is that one token is about four characters, or about three-quarters of a word. Our toy manages only … characters per token on your text.
Two quirks are real, too. Capitals matter: “▁Puff” is one token here but “▁puff” is several, because the stories always write the name with a capital. And the space belongs to the word after it, so the “cat” at the end of “thundercat” is not the same token as “▁cat”.
Sources
- R. Sennrich, B. Haddow and A. Birch, Neural Machine Translation of Rare Words with Subword Units, ACL 2016.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever, Language Models are Unsupervised Multitask Learners, OpenAI, 2019 (the GPT-2 paper; byte-level BPE and the 50,257-token vocabulary).
- P. Gage, A New Algorithm for Data Compression, The C Users Journal, February 1994.
- OpenAI Help Center, What are tokens and how to count them? (the four-characters rule of thumb).