Chapter 5 · From model to chatbotPuff 20 of 23

Learning to be helpful

A freshly pretrained model has read half the internet, but it isn’t an assistant. Ask it a question and it might reply with three more questions, because that’s what forums often do next. So how does a text predictor become something that actually answers you?

Guess first

A model trained only to predict internet text is asked a question. Why might it not answer it?

Play with it

Fig. 1 · Before and after Same question, two models

You typeHow do I keep basil alive on a windowsill?

Pretrained only

How do I keep basil alive on a windowsill? How do I stop mint taking over the pot? Why are my tomato leaves going yellow? These are the questions we hear most at the garden centre, so this month’s newsletter tackles all three…

After fine-tuning

Give it your sunniest window, water it when the top of the soil feels dry, and pinch off any flower buds so it keeps making leaves.

Illustrative: written for this page to show the typical difference, not real model output

The first fix is simple: show the model thousands of example conversations written by people, and keep training it on the same next-word game. Then comes a subtler step. Below, you are the person judging answers, and a tiny reward model (a scorer that rates answers) learns your taste from your picks.

Fig. 2 · Teaching by preference You pick, the reward model learns

Someone asked:…

What the reward model has learned

dislikeslikes

The chatbot answers a new question

Asked “How many minutes are in 3 hours?”, it scores four drafts and keeps the best:

Tap the better answer in each pair. The answers and their scores were written for this page; the reward model’s learning is real and runs in your browser.

How it works

Tap a step to see it in the figure.

Check yourself

What can go wrong if the reward model happens to love long answers?

Go deeper: the mathThe Bradley–Terry reward model, and how real systems use it

A score for every answer

The page’s reward model gives each answer a score r: a weighted sum of its four features (how well it answers, whether it’s correct, how friendly it is, and its length in words ÷ 15).

r(answer) = w₁·answers + w₂·correct + w₃·friendly + w₄·length

Right now: no picks yet, so every weight is 0.

From scores to preferences

The Bradley–Terry model turns two scores into the chance that a person prefers one answer, using the S-shaped sigmoid σ from the neuron puff:

P(A preferred to B) = σ(r(A) − r(B)) = 11 + e^{−(r(A) − r(B))}

For the pair on screen: …

Learning the weights

Learning means making your actual picks as likely as possible. The loss is the average surprise at each pick, plus a small penalty that keeps the weights from growing without limit when the picks are easy to explain:

L = − 1n Σ log σ(r(chosen) − r(rejected)) + λ2 Σ w_k²

Gradient descent nudges the four weights downhill on L, exactly as in chapter 1. Here λ = 0.05; the last fit took no steps yet.

How real systems do it

In the InstructGPT recipe (Ouyang and colleagues, 2022), people first wrote example answers for supervised fine-tuning, then ranked several drafts per prompt. The reward model was itself a large language model that reads the whole answer and outputs one number; there are no hand-picked features. The chatbot was then trained with reinforcement learning to write high-scoring answers, with a penalty for drifting too far from its fine-tuned self. People preferred the answers of a 1.3-billion-knob InstructGPT to those of the 175-billion-knob GPT-3 it started from. The idea of learning a reward from pairwise human choices goes back to Christiano and colleagues (2017). This page uses a simpler, also real, trick called best-of-n: draft several answers and keep the top-scoring one.

Pushing too hard on a learned reward backfires. Gao, Schulman and Hilton (2022) measured how the true quality of answers first rises, then falls, as a chatbot over-optimises a reward model, which is why real training holds the chatbot back from straying too far. Newer methods skip parts of the recipe: Direct Preference Optimisation tunes the chatbot directly on the preference pairs with no separate reward model (Rafailov and colleagues, 2023), and Constitutional AI has an AI give the feedback, guided by a written list of principles (Bai and colleagues, 2022).

Sources

L. Ouyang et al., Training language models to follow instructions with human feedback, 2022.
P. Christiano et al., Deep reinforcement learning from human preferences, 2017.
L. Gao, J. Schulman and J. Hilton, Scaling Laws for Reward Model Overoptimization, 2022.
R. Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023; Y. Bai et al., Constitutional AI: Harmlessness from AI Feedback, 2022.
The questions, answers and feature scores on this page are made up for illustration.