Chapter 3 · Words as numbersPuff 11 of 23
Words as places on a map
“Puppy” and “puppet” differ by two letters. “Puppy” and “dog” share none. Yet a chatbot treats puppy and dog as close cousins and puppet as a stranger. By the time it starts thinking, every token has been swapped for a number. How can a number know what a word means?
Guess first
On a map of meanings, which word should land closest to “puppy”?
Play with it
Word arithmetic
Each dot is a word, placed by two of its seven numbers. Pick a word (or tap a dot) to see its nearest neighbours, measured over all seven numbers. Change the two axes to look at the map from another side.
How it works
Tap a step to see it in the figure.
Check yourself
Why are “puppy” and “puppet” far apart on the map?
Go deeper: the mathVectors, cosine similarity and word arithmetic
A word is a vector
A list of numbers is a vector. On this page each word is a vector of seven numbers, in the order animal, person, object, royal, male ↔ female, young, big. For example:
puppy = (1, 0, 0, 0, 0, 1, 0.2)
Real models use hundreds or thousands of numbers per word (or per token), and nobody chooses what they mean.
Cosine similarity
Multiply two vectors number by number and add up the results: that is the dot product. Divide by both lengths and you get the cosine of the angle between them:
similarity(a, b) = a₁b₁ + a₂b₂ + … + a₇b₇|a| × |b| where |a| = √(a₁² + … + a₇²)
Right now, for the chosen word and its nearest neighbour:
…
It is 1 (100%) when two vectors point exactly the same way and 0 when they share nothing. Cosine ignores how long the vectors are and looks only at their direction, which is the usual way to compare word vectors.
Word arithmetic
Subtract and add vectors number by number, then look for the word whose vector is most similar to the result, leaving out the three words you started with:
…
That last rule matters. If the three input words were allowed, the nearest word would be ….
Here the sums land neatly because the numbers were typed in by hand to be tidy. Learned vectors are messier. Nissim, van Noord and van der Goot (2020) showed that many famous analogy results depend on leaving out the input words and on choosing examples that happen to work. Try crown − king + queen: the sensible answer, crown, is one of the inputs and is ruled out, so the sum lands somewhere silly.
Where real vectors come from
In 1957 the linguist J. R. Firth wrote, “You shall know a word by the company it keeps” (A Synopsis of Linguistic Theory, 1930–1955). Word2vec (Mikolov and colleagues, 2013) turned that idea into a learning task: a small network nudges every word’s numbers so that it gets better at predicting the words around it, exactly the nudging from chapter 1. Words used in similar company end up with similar numbers. Nobody labels the directions; a direction like “royalty” is something people may spot afterwards, if it is there at all.
A chatbot does the same for every token: a big table gives each token ID its own vector, and those numbers are learned along with the rest of the model.
Sources
- T. Mikolov, K. Chen, G. Corrado and J. Dean, Efficient Estimation of Word Representations in Vector Space, 2013.
- M. Nissim, R. van Noord and R. van der Goot, Fair is Better than Sensational: Man is to Doctor as Woman is to Doctor, Computational Linguistics, 2020.
- D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd edition draft, chapter “Vector Semantics and Embeddings”.
- The seven numbers for each word were made up by hand for this page. Real word vectors are learned from text, much longer, and unlabelled.