1.3 — How Models Generate Text: Sampling, Temperature, Top-p

Learning objective: describe how a model chooses its next token and how sampling parameters change output behavior.

At each step, a model doesn’t “know” the next word — it computes a probability distribution over its entire vocabulary for what token comes next, then samples from that distribution. Several parameters control how that sampling happens:

  • Temperature scales how “confident” vs. “spread out” the distribution is before sampling. Temperature 0 (or near it) makes the model pick the highest-probability token almost every time — deterministic, repeatable, good for factual/structured tasks. Higher temperature flattens the distribution, giving lower-probability tokens a real chance of being picked — more variety, more creativity, more risk of going off the rails.
  • Top-p (nucleus sampling) restricts sampling to the smallest set of tokens whose cumulative probability exceeds p, cutting off the long tail of very unlikely tokens regardless of temperature.
  • Top-k is a simpler cousin: only consider the k most likely tokens.

A worked example

Imagine the model is completing “The weather today is” and its raw probability distribution over the next token looks like this:

flowchart LR
    A["'The weather today is ___'"] --> B{Probability distribution}
    B -->|"38%"| C["sunny"]
    B -->|"24%"| D["cloudy"]
    B -->|"15%"| E["cold"]
    B -->|"9%"| F["nice"]
    B -->|"6%"| G["terrible"]
    B -->|"8%"| H["...long tail"]
  • At temperature 0, the model picks “sunny” essentially every time — the highest-probability option, deterministically.
  • At higher temperature, the distribution flattens, so “cloudy,” “cold,” or even “terrible” become live possibilities — more variety, but also more chance of drifting into an odd continuation.
  • Top-p = 0.9 would restrict sampling to just enough of the highest-probability tokens to cover 90% of the distribution — likely “sunny,” “cloudy,” “cold,” and “nice” — cutting off the long tail regardless of temperature.

What this looks like as API parameters

response = client.messages.create(
    model="example-model",
    max_tokens=200,
    temperature=0,       # deterministic — good for extraction, classification
    messages=[{"role": "user", "content": "Extract the invoice total as JSON."}],
)

response_creative = client.messages.create(
    model="example-model",
    max_tokens=200,
    temperature=1.0,     # more variety — good for brainstorming, creative writing
    messages=[{"role": "user", "content": "Suggest five taglines for a coffee shop."}],
)

The task should decide the setting: deterministic tasks with one correct answer want low temperature; open-ended generation wants room to vary.

The security-relevant misconception

Low or zero temperature does not make a model safe or deterministic against adversarial input. It makes output more reproducible for a fixed prompt, but an attacker crafting adversarial input is choosing the prompt, not relying on random sampling — this is a common misconception worth correcting early, and it foreshadows Track 3. A temperature-0 model given a cleverly crafted jailbreak prompt will reproduce the same successful jailbreak deterministically, every time — determinism helps the attacker iterate reliably, it doesn’t protect you.

Exercise

Run the same prompt three times at temperature 0 and three times at temperature 1 (using any model playground). Compare variability within each group. Then write one sentence on why you would or wouldn’t use temperature 0 for a customer-facing support bot vs. a creative writing tool.

← Context Windows · Back to Module 1 · Next: Base vs. Instruction-Tuned →