Chapter 5 · From model to chatbotPuff 22 of 23
Why it makes things up
You ask a chatbot when a small local museum opened. It answers at once, in a calm, sure voice, with a precise year. The year is wrong. Nobody hacked it, and it wasn’t trying to fool you. So where did that confident answer come from?
Guess first
A chatbot confidently gives a wrong date for a historical event. What most likely happened?
Play with it
The capital of France is ?
Puff says…
… The fact check comes from this page, not from the model. The model has no idea which answers are true.Pick a country. The model is a word-counting trigram (puff 12) trained on a few sentences written for this page; you can read them all in “Go deeper”.
How it works
Tap a step to see it in the figure.
Check yourself
Which of these helps most against made-up facts?
Go deeper: the mathBackoff, why probability isn’t truth, and calibration
Everything this model has read
The capital of France is Paris. The biggest city in France is Paris. The capital of Italy is Rome. The capital of Japan is Tokyo. The capital of Egypt is Cairo. The capital of Spain is Madrid. The capital of Germany is Berlin. The capital of Kenya is Nairobi. The capital of Canada is Ottawa. The capital of Greece is Athens. The capital of Portugal is Lisbon. The Sun is a star. Bees make honey. Spiders have eight legs.
Backoff
A trigram predicts the next word w from the last two words u v. When that pair never appeared, the page’s model backs off to the last word alone, and if needed to no context at all:
p(w | u v) = count(u v w)count(u v) or else count(v w)count(v) or else count(w)N
Right now: …
A neural network doesn’t switch between tables like this, but it does something similar in spirit: for a context it has rarely seen, its guess is shaped by similar contexts it has seen. Facts that appeared only once or twice in the training text are the ones most likely to be filled in from the pattern instead.
Why probability isn’t truth
Training rewards one thing: giving high probability to the words that really came next in the text. It never checks the world. Most of the time, likely and true go together, because most text is roughly right. But the model must always put its probability somewhere, and when it has no fact, the pattern wins. Kalai and colleagues (2025) add that tests and leaderboards usually score “I don’t know” as zero, just like a wrong answer, so a model that always guesses scores better than one that admits doubt. Like a student on a multiple-choice exam, it learns to bluff.
Calibration
A model is calibrated if its confidence matches its accuracy: of all the answers it gives with 70% probability, about 70% should be right. This page’s model actually carries a useful signal: 100% for France, much less for Peru. The trouble is that the sentence it writes throws that number away. One practical check builds on this: ask the same question several times, as “Ask again” does. Answers the model knows stay the same; made-up ones tend to change (Manakul and colleagues, 2023).
Sources
- A. T. Kalai, O. Nachum, S. S. Vempala and E. Zhang, Why Language Models Hallucinate, 2025.
- Z. Ji et al., Survey of Hallucination in Natural Language Generation, 2023.
- P. Manakul, A. Liusie and M. J. F. Gales, SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models, 2023.
- D. Jurafsky and J. H. Martin, Speech and Language Processing, 3rd edition draft, chapter 3: n-gram language models and backoff.
- The training sentences are written for this page. The fact check uses each country’s real capital.