Tokens, context and generation

Before you can reason about an LLM application's cost, speed, or limits, you need the units the model actually works in. It does not read words or sentences; it reads and writes tokens, inside a finite context window, one token at a time. Almost every practical constraint in this book follows from those three facts.

What you will understand by the end

• What a token is, and why cost and limits are counted in tokens, not words.
• What the context window is, and why it's a budget you manage.
• How generation works (prompt in, one token out at a time) — and why the same input can yield different outputs.
• Why temperature 0 is not determinism, and what to do instead when you need reproducibility.

Tokens

Models don't operate on characters or words; text is first split into tokens — sub-word chunks. Common words may be one token; rarer words split into several. A useful rough anchor for English is ~4 characters per token, or ~0.75 words per token, but it varies by tokenizer and language.

Two practical consequences: counting tokens (not words) is how you predict cost and whether input fits; and non-English or code-heavy text can tokenize very differently, so never assume a word count.

The context window

Everything the model can "see" for one request — the system prompt, retrieved context, the conversation so far, the user's input, and the space reserved for its output — must fit inside a fixed context window measured in tokens.

This is why applications do things like retrieve only the most relevant context, summarise long histories, and cap output length — they are managing a finite token budget. Bigger windows exist, but a larger window is capacity planning, not a free pass: more tokens cost more and can be slower.

Generation: one token at a time

An LLM generates autoregressively — it produces the next token, appends it, and repeats, each token conditioned on everything before it. That splits a request into two phases you'll meet again in depth in Prefill and decode:

• Reading the prompt (prefill): the whole input is processed to set up state.
• Writing the answer (decode): tokens are emitted one at a time until the model stops or hits the output cap.

Why the same input gives different outputs

At each step the model has a probability distribution over possible next tokens. How it picks — the sampling — is controlled by settings like temperature:

• Temperature 0 (greedy): take the highest-probability token every time → the most repeatable output available to you.
• Higher temperature: sample more randomly → more variety and creativity, but less repeatable.

That is a real dial, and where a model exposes it, turning it to 0 is the right move when you want repeatability. But it is easy to draw the wrong conclusion from it, so be precise about what it buys — and check that the model still offers it at all.

Temperature 0 is not determinism

Greedy decoding removes the deliberate randomness in how a token is chosen. It does nothing about the numbers that choice is made from. Send an identical request twice to a hosted model and the logits need not come back bit-identical — not because anything is random, but because the computation isn't guaranteed to execute identically, and because the system on the other end can change.

Providers generally document only the conclusion — that temperature 0 is not fully deterministic — rather than the mechanism. The usual engineering explanations, offered here as plausible causes rather than confirmed ones:

• Numerically non-identical execution. Floating-point addition isn't associative, so a kernel that sums in a different order returns slightly different values. Batch shape, kernel selection, and GPU generation can all change that order. Usually irrelevant; occasionally enough to flip which of two near-tied tokens wins — and that flip then conditions every token after it.
• The system behind the name. A version alias can be re-pointed and a serving stack upgraded without the model string you send ever changing.

Either way, the observable pattern is the same: output that is stable almost always, and then quietly isn't. The greedy choice is only fragile where the top two candidates are nearly tied — rare per token, but you generate a great many tokens.

On current frontier models, you may not have the dial

There is a further wrinkle that makes "just set temperature 0" not merely incomplete but, on some models, impossible. Newer Claude models no longer let applications control sampling: a non-default temperature, top_p, or top_k returns a 400 on Claude Opus 5, Opus 4.8, and Opus 4.7, and on Claude Sonnet 5. Since the default temperature isn't 0, temperature: 0 is exactly the kind of value that gets rejected. Omit the parameters on those models and steer through the prompt and reasoning-effort controls instead. Older models — Opus 4.6, Sonnet 4.6, Haiku 4.5 — still accept them.

So an evaluation harness that hard-codes temperature: 0 works on models that still support the control and fails on newer ones that reject it — with a 400 rather than a silent quality change. Treat decoding parameters as provider- and version-specific, not as universal knobs — and note what this does to the argument above: on a model that won't take the parameter, repeatability has to be characterised under whatever defaults it does use. Measuring it stops being good practice and becomes the only option.

Measure reproducibility at the level you score

"How often does the output change?" is the wrong question on its own, because it can be both too strict and too lenient. Two responses can differ in whitespace or phrasing while carrying the identical answer; two responses can be nearly identical text while one has a wrong value in a field you grade on. Pick the level that matches your evaluation contract:

• Exact output. Byte-for-byte agreement. The strictest and usually the least informative — it flags harmless variation.
• Semantic or structured agreement. Does the parsed object — the extracted fields, the query intent, the chosen label — match? This is the right level for most structured tasks.
• Score variance. Does the task's pass/fail, or the aggregate metric, move? This is what actually threatens a conclusion, and it's the number to report.

These can diverge sharply, and the gap is itself informative: high text variance with zero score variance means your scoring is appropriately tolerant, while low text variance with flipping pass/fail means your grader is balanced on a knife edge.

Mental model

The model reads and writes in tokens, inside a finite window, one token at a time. Cost, latency, and context limits all follow from that. Repeatability doesn't: greedy decoding — where a model still offers it — removes randomized token selection and leaves every other source of variation in place, so reproducibility is something you pin down, measure at the level you score, and report — not something you set.

Common mistakes

• Estimating length in words. Token counts drive cost and limits; word counts mislead, especially for code or non-English text.
• Forgetting output shares the window. Sizing the prompt to the window leaves no room to answer.
• Equating temperature 0 with determinism. It removes randomized token selection and nothing beneath it; identical requests can still diverge.
• Assuming reproducibility instead of measuring it. If you never re-ran the same input, you don't know how much it moves — you're guessing it's zero.
• Measuring reproducibility at the wrong level. Byte-level diffs flag harmless variation and can still miss a field change that flips a score.
• Calling a model by a moving alias in an experiment. The name can keep pointing at something new; pin the version, or your baseline shifts underneath you.
• Hard-coding non-default decoding parameters as if they were universal. They are provider- and version-specific, and can make requests fail on newer models that reject them.
• Treating a bigger context window as free. More tokens cost more and can be slower; size context to what's needed.

Practical guidance

• Count tokens (with the model's tokenizer) when budgeting cost or checking fit.
• Reserve output headroom: keep prompt + expected output comfortably under the window.
• Where sampling parameters are supported, set temperature 0 for evaluation work — useful, just not sufficient. Where they aren't, characterise repeatability under the model's own defaults instead.
• Pin the exact model version, not a floating alias, and record the whole request as a manifest: model version, prompt version, schema, decoding settings, and the date.
• Repeat trials and report the variance, measured at the level you score — usually structured agreement and score movement, not byte-level text diffs. A measured number is evidence; an assumed one isn't.
• Scope every invariance claim to the model, deployment, dataset, and load range you actually tested. "Bit-identical across this concurrency sweep" is a result; "deterministic at temperature 0" is a generalisation you haven't earned.
• Keep decoding settings behind a per-model configuration rather than hard-coded, so a model that rejects them doesn't take the harness down.
• Manage context deliberately — retrieve the most relevant, summarise long histories, cap output length.

Summary

• Text becomes tokens; cost, limits, and latency are all counted in them.
• The context window is a shared token budget for input and output.
• Generation is autoregressive and sampled, so output length drives latency and the same input can vary.
• Temperature 0 is greedy decoding, not determinism — it removes randomized token selection, while non-identical execution and silent version changes sit below it.
• Newer Claude models reject non-default sampling parameters, so treat decoding settings as version-specific rather than universal.
• Measure reproducibility at the level you score — structured agreement and score movement, not just byte-level text diffs.
• Reproducibility is pinned, measured, reported, and scoped — never assumed.

Knowledge check

A 3,700-token prompt goes to a model with a 4,096-token window, and you ask for up to 512 output tokens. What happens?

It overflows: 3,700 + 512 = 4,212 > 4,096. Input and output share the window, so you'd get an error (or a truncated answer). You'd need a shorter prompt, a smaller output cap, or a larger window. This exact arithmetic is a common source of "why did my response get cut off?".

Your extraction task must be reproducible for evaluation. What sampling setting do you use — and is that enough?

Temperature 0 (greedy) where the model supports it: it takes the highest-probability token at each step, removing randomized selection as a source of variation. That's the right setting, and it is not enough — and on some current models you can't set it at all, since newer Claude models reject a non-default temperature with a 400.

Greedy decoding fixes how a token is chosen, not the numbers it chooses from. Execution that isn't bit-identical, or a silently-updated model behind a stable alias, can shift those numbers enough to flip a near-tied token. So also: pin the exact model version rather than an alias, record the full request as a manifest, and re-run the same inputs and measure how much the result moves — at the level you actually score. For an extraction task that means comparing the parsed objects and the pass/fail outcome, not the raw text: identical extractions can be worded differently, and near-identical text can carry a wrong field.

You re-run your 200-task eval on the identical config and get 94.5% instead of yesterday's 95.0%. Nothing in your code changed. What are the candidate explanations, and how do you tell them apart?

One task changed hands (200 tasks, 0.5 points). Three hypotheses — and you have no basis yet for ranking them, which is itself the finding:

• Ordinary run-to-run variation — a near-tied token flipped. Possible even at temperature 0.
• The model changed underneath you — if you called a floating alias rather than a pinned version, you may not be measuring the same system at all.
• Something in the harness is not as fixed as you think — a timestamp in the prompt, a reordered schema, a retry that fired only on one run.

You tell them apart with two things you should already have: a manifest to diff (which settles the second and third), and a measured variance baseline (which tells you whether 0.5 points is inside normal movement for this setup). Without both, you can only guess which one you're looking at — you can't even say the first is the likely explanation, because you never established what normal looks like.

That's the practical cost of treating "we ran at temperature 0" as a reproducibility claim. And there's a sharper problem underneath: on a single run per task you cannot distinguish a real regression from a re-roll at all. See Prompt and model comparisons.

Related chapters

• Prompts, schemas and structured outputs — how you steer what those tokens say
• Prefill and decode — the same generation, from the performance side
• Sample size and uncertainty — why one run per task understates uncertainty
• Prompt and model comparisons — telling a real change from a re-roll