Sampling, temperature and top-k

Sampling is how a language model picks its next token from its probabilities. Temperature and top-k are the settings that shape that choice. They are why the same prompt can give different text each time.

The model's numbers do not change between two answers. Only the choice does.

After "the bus is", our invented model gives scores for four next words. Softmax turns them into probabilities.

Greedy decoding always takes the most likely token, here late. It gives the same text every time. Over a long text it often gets stuck repeating itself, because the likeliest next step can lead back to the same place.

Sampling draws a token at random, each with its probability. So late comes out about 64 times in 100, and blue about 3.

Temperature changes those chances first. Divide every score by it, then apply softmax. Below 1, the top token gets even more likely. Above 1, the chances even out and rarer words appear more often.

docker exec jupyter python3 -c "import numpy as np; s = np.array([3, 2, 1, 0]); print([np.bincount(np.random.default_rng(7).choice(4, size=1000, p=np.exp(s / t) / np.exp(s / t).sum()), minlength=4).tolist() for t in (1.0, 0.5)])"

Top-k keeps only the k most likely tokens and shares their chances out again so they add up to 1. With k = 2, here and blue can never appear.

A random draw starts from a seed. The same seed gives the same draws, and a new seed gives new ones. That is why one prompt can give different text, and why a fixed seed makes a test repeat.

import math
import random

words = ["late", "full", "here", "blue"]
scores = [3, 2, 1, 0]      # the model's scores after "the bus is"
temperature = 1.0
seed = 1

powers = []
for s in scores:
    powers.append(math.exp(s / temperature))
total = sum(powers)
probs = []
for p in powers:
    probs.append(p / total)
print("probabilities:", [round(p, 3) for p in probs])

random.seed(seed)
draws = random.choices(words, weights=probs, k=8)
print("eight draws:", draws)
# Change seed to 2 and press Run: different draws, same probabilities.
# Then set temperature to 0.3. What happens to the draws?