Chapter 4 · The transformerPuff 15 of 23

Questions, labels and answers

“Puff saw the big grey cloud.” By the time the model reaches “cloud”, it should know the cloud is big and grey. In the last puff, each word looked back and chose which words to listen to. But how does “cloud” know that “big” and “grey” are the words describing it, and what exactly does it get from them?

Guess first

Attention gives every word three lists of numbers, not one. Why three?

Play with it

Fig. 1 · Queries, keys and values One word asks, the others answer

Toy model: hand-made lists of 3 numbers. A query and a key measure thing, describer and action; a value carries character, size and colour. Real models learn lists of dozens of numbers that no one has named. Every score, share and blend here is computed live.

“cloud” is asking. Drag the pink sliders to change what it is looking for, or tap another word to make it ask.

How it works

Tap a step to see it in the figure.

Check yourself

A word’s key matches the query perfectly, but its value is all zeros. What does the asking word receive from it?

Go deeper: the mathWhere the three lists come from, and the sum with live numbers

Three lists from one

In a real model, a word arrives as one long list of numbers, its embedding x. Three tables of learned knobs (matrices) turn it into the three lists:

q = W_Q x
k = W_K x
v = W_V x

Multiplying a list by a matrix makes each new number a weighted sum of all the old ones. Training adjusts W_Q, W_K and W_V like any other knobs, so the model learns what to ask, how to label itself and what to pass on. This page skips that step and hand-makes q, k and v directly.

Scaled dot-product attention

For one asking word with query q, and every word j it can see:

score_j = q · k_j√d
share_j = softmax(score)_j
output = Σ_j share_j v_j

Here d = 3. With the query [0, 2, 0] from “cloud”:

Each word’s key and value are the lists shown in the figure.

Adding up the last column gives the output, … (character, size, colour). Doing every word at once is the same sum written with tables: softmax(QK^T/√d)V.

Why split the jobs?

With a single list per word, a word’s question would also be its label, so it would mostly find itself and words like it. With separate lists, “cloud” can ask for describing words while being labelled as a thing, and “saw” can be easy to find while carrying nothing in this toy’s three value slots.

Real models also add the causal mask from the last puff (an asking word cannot see later words), and run many of these attention units side by side. That’s the next puff.

Sources

A. Vaswani et al., “Attention Is All You Need”, 2017. Section 3.2 defines queries, keys, values and scaled dot-product attention.
J. Alammar, “The Illustrated Transformer”, 2018. A friendly picture-by-picture walk through the same sums.
The three-number lists in the figure are hand-made for this page; the formula is the real one.