Chapter 5 · From model to chatbotPuff 21 of 23

How much it can remember

At the start of a long chat, Asha mentions she’s allergic to peanuts. Twenty minutes later she asks whether she can eat the satay. A friend would remember. Will the chatbot?

Guess first

In a very long chat, a chatbot forgets what you said at the start. Why?

Play with it

Fig. 1 · The context window What the model can see when Asha asks

…

Puff’s reply (scripted for this page)…

The chat is an example written for this page. Slide the window: tokens outside it are greyed out, because the model cannot see them at all when it writes its reply.

Tokens here are counted roughly: every word and every punctuation mark is one token. Real tokenisers (puff 10) split rarer words into pieces, so real counts are a little higher. Real windows are also far bigger than this one; it is shrunk so the whole chat fits on the screen.

Fig. 2 · The cost of a window Every token looks back at every earlier one

Each row is one token. Its blue squares are the tokens it compares itself with in attention: itself and everything before it. Double the window and watch the count.

How it works

Tap a step to see it in the figures.

Check yourself

Doubling the context window roughly does what to the work attention has to do?

Go deeper: the mathCounting the pairs, real window sizes, and workarounds

A triangle of pairs

In a chatbot, each token may look only at itself and the tokens before it (the causal mask: it can’t peek at words not yet written). Token 1 has 1 pair, token 2 has 2, and token n has n. Adding them up:

pairs = 1 + 2 + … + n = n(n + 1)2 ≈ n²2

In Fig. 2 right now: …. That count is for one attention head in one layer; real models have dozens of each. Without the mask, as in models that read a whole text at once, it would be the full square, n².

Real window sizes

GPT-2 (2019) had a window of 1,024 tokens and GPT-3 (2020) one of 2,048. Google reported that its Gemini 1.5 models could find details in contexts of millions of tokens, up to at least 10 million in tests. The pair count shows why that took engineering: …

A big window isn’t perfect recall, either. Liu and colleagues (2023) found that models used facts placed at the start or end of a long context much more reliably than facts buried in the middle, an effect they called “lost in the middle”.

Workarounds

Summaries: squeeze old messages into a short summary that stays in the window.
Notes: save key facts (“Asha is allergic to peanuts”) and paste them back in each time.
Retrieval: keep documents outside the model, search them for passages related to the question, and paste just those into the window. Lewis and colleagues (2020) called this retrieval-augmented generation.

What this page simplifies

Token counts use a word-and-punctuation split, and real chat formats add hidden marker tokens between messages. Puff’s two replies are scripted, not computed. And attention’s pairs are not the only cost: the model also has to store numbers for every token in the window, which grows in step with n.

Sources

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei and I. Sutskever, Language Models are Unsupervised Multitask Learners, OpenAI, 2019 (GPT-2’s 1,024-token window); T. B. Brown et al., Language Models are Few-Shot Learners, 2020 (GPT-3’s 2,048).
Gemini Team, Google, Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
N. F. Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023.
P. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020.
The chat and Puff’s replies are made up for this page.