Optional Foundation — Build a Small Language Model

From corpus to training examples

Data and parameters·Advanced·7 min read

Now flip the book over. You have spent the inference chapters running a pile of trained weights; this part shows where those weights come from — starting with the one idea that makes training possible: a raw text corpus turns itself into billions of labelled prediction tasks, for free.

Every chapter so far has treated the model's weights as a given — a fixed object you load, quantize, cache, and serve. This part turns the book upside down and asks the obvious next question: where did those weights come from? The answer is training, and training starts not with the model but with the data — and with a trick that turns ordinary text into an endless supply of labelled examples without anyone labelling anything.

What you will understand by the end

  • Why a language model is trained on next-token prediction, and why that is self-supervised (no human labels needed).
  • How a stream of text becomes input → target training pairs by shifting one position.
  • Why data quality, not just quantity, sets the ceiling on the model — the "small model, narrow data" idea.
  • The plumbing — train/validation split, batching, context window — that feeds the loop.

The objective: predict the next token

Recall from Tokens, context and generation that a model reads and writes tokens, one at a time. Training is just teaching it to do that one thing well: given the tokens so far, predict the next one. That is the entire objective. Everything a language model appears to "know" is a side effect of getting very good at this single game over a huge amount of text.

The beautiful part is that the game labels itself. You do not need annotators to say what the right answer is — the text already contains it. For any position in any document, the "correct next token" is simply the token that actually comes next. This is what self-supervised means: the supervision signal is manufactured from the raw data.

Key idea

Next-token prediction is a self-supervised task: the label is the next token, which the corpus already provides. That is why language models can train on trillions of tokens of ordinary text — every document is a pile of free, pre-labelled examples. No annotation step exists.

From text to training pairs: shift by one

Concretely, you take a chunk of tokenised text of length equal to the context window, and build a training pair by copying it and shifting one position:

  tokens:   [ t1  t2  t3  t4  t5  t6 ]

  input  X = [ t1  t2  t3  t4  t5 ]
  target Y = [ t2  t3  t4  t5  t6 ]
             └ each Y is "the next token" for the matching X position ┘

Read column by column: given t1, predict t2; given t1 t2, predict t3; and so on. A single length-C sequence therefore encodes C separate prediction tasks at once — the model is scored on every position in parallel. That "shift the sequence by one to make the target" is the mechanical heart of language-model training.

Key idea

The target is just the input shifted left by one token. This is why one training sequence teaches the model many predictions simultaneously, and why the same autoregressive structure you met in inference (each token conditioned on all before it) shows up in training — training and generation are the same shape, run in opposite directions.

Data quality is the real lever

It is tempting to think a bigger pile of text is all that matters. But the corpus is what the model becomes — its knowledge, style, and failure modes are inherited from the data. A famous demonstration of this is training a small model (tens of millions of parameters, not billions) on a deliberately narrow, clean dataset — for example, simple children's stories written with a tiny vocabulary. Such a model, far too small to be a general assistant, can still produce fluent, grammatical, coherent text in that domain, because the data it learned from was consistent and high quality.

Watch out

"More tokens" is not the same as "better model." Noisy, contradictory, or off-domain data teaches noisy, contradictory behaviour. For a small model especially, a narrow, high-quality corpus beats a huge messy one — the data sets the ceiling, and no amount of training or clever architecture recovers what the corpus never contained.

The plumbing that feeds the loop

Two more pieces turn a corpus into something a training loop can consume:

  • Train / validation split. Hold out a slice of the data (say 10–20%) that the model never trains on, so you can measure whether it is genuinely learning the language or just memorising the training set. You will use this split to read the validation loss in the optimization chapter.
  • Batches and the context window. The data is chopped into fixed-length sequences (the context window) and grouped into batches so the GPU processes many sequences at once — the same batching logic that makes inference efficient applies to training. Tokenisation itself is the same subword scheme you already know; training simply consumes the token IDs it produces.

Mental model

A corpus is not raw material waiting to be labelled — it is already labelled. Shift the tokens by one and every document becomes a stack of "predict the next token" tasks. The quality of that corpus is the quality of the model; the split, batches, and context window are just how you feed it.

Common mistakes

  • Thinking training needs labelled data. Next-token prediction is self-supervised — the label is the next token, free from the text itself.
  • Chasing token count over quality. A narrow, clean corpus can beat a huge noisy one, especially at small scale; the data is the ceiling.
  • Skipping the validation split. Without held-out data you cannot tell learning from memorising — the split is what makes the loss trustworthy.
  • Forgetting one sequence = many tasks. Every position in a context window is a scored prediction, not just the last one.

Practical guidance

  • Treat dataset construction as a real engineering step, not a prerequisite — curate, clean, and dedupe; the payoff shows up as model quality you cannot get any other way.
  • Pick a context window that fits the structure of your text and your memory budget; it sets both how much history each prediction sees and the KV-cache cost at inference.
  • Always carve out a held-out validation slice before training so the loss curves later mean something.
  • Reuse your inference tokenizer for training so token IDs match end to end.

Summary

  • Language models are trained on next-token prediction, a self-supervised task whose labels come free from the corpus.
  • You build training pairs by shifting the tokens one position (input → target); one sequence encodes many predictions.
  • Data quality sets the ceiling — a small model on narrow, clean data can be fluent in its domain.
  • A train/validation split, batches, and a context window are the plumbing that turns a corpus into a training loop's input.

Knowledge check

Why is next-token prediction called "self-supervised," and what does that let you do that supervised learning can't?

Because the label — the correct next token — is taken directly from the text itself, so no human annotation is required. That lets you train on essentially unlimited raw text (every document is pre-labelled), which is what makes training on trillions of tokens feasible; classic supervised learning would need each of those examples hand-labelled.

You have a length-256 context window. From one such sequence, how many next-token prediction tasks does the model train on, and how is the target built?

256 (one per position, though the first has no history and the standard construction gives you the 255 "predict position i+1 from positions 1..i" tasks per sequence). The target is the input shifted left by one token: at each position the model is scored on predicting the token that actually follows, so a single sequence supplies many labelled tasks at once.

Related chapters