Why it answers differently each time

Temperature

A neural network doesn’t output probabilities directly. It outputs a score z_i for every token i (scores can be any number), and the softmax function turns them into probabilities. Temperature T divides the scores first:

p_i = e^{z_i / T}Σ_j e^{z_j / T}

Our word-counting model has probabilities rather than scores, but taking z_i = ln p_i (the natural logarithm) gives exactly the same reshaping: raise each chance to the power 1/T, then divide by the total so they add up to 1 again.

…

At T = 1 nothing changes. Below 1, big chances grow and small ones shrink. As T goes to 0, all the chance piles onto the top word, so the page treats 0 as “always take the top word”.

Top-k and top-p

Two common extra rules cut off the long tail of unlikely words before rolling. Top-k keeps only the k likeliest words. Top-p, also called nucleus sampling, keeps the smallest set of top words whose chances add up to at least p. Either way, the kept chances are scaled up to add to 1. Right now:

…

Holtzman and colleagues (2019), who proposed top-p, showed that always taking the top word makes text dull and repetitive, while sampling from the full list lets in too many odd words. Try “the cat” at temperature 0 to see the dull, repetitive kind.

What this page simplifies

This model only knows words that actually followed the prompt’s last two words in the stories, so even a high temperature only shuffles between sensible-ish options. A real chatbot gives every one of its tokens some chance, so a high enough temperature makes it write nonsense.

And temperature 0 is not always perfectly repeatable in real systems. Big models run on hardware that adds up huge lists of numbers in parallel, and tiny rounding differences can occasionally flip a near-tie between two words. On this page, 0 means identical every time.