Optional Foundation — Build a Small Language Model

From trained weights to generation — and what fine-tuning adds

Training and generation·Advanced·7 min read

Close the loop. Once training converges you have a set of weights — and generating from them is exactly the inference you already know. Then mark the boundary this part deliberately drew: what pre-training gives you, and what fine-tuning, instruction-tuning and RAG add on top.

The training loop converges, the loss flattens, and you are left with the one object this whole book started from: a set of fixed weights. This final chapter closes the circle. Generating text from those weights is not a new topic — it is precisely the inference you spent the first parts of the book on, now seen from the far side. Then we name the boundary this part drew on purpose: what pre-training actually produced, and what the next stages — fine-tuning, instruction-tuning, RAG — add.

What you will understand by the end

  • Why generation from a trained model is just the inference loop you already know.
  • The exact moment training's two-directional view collapses into one-token-at-a-time decode.
  • What pre-training gives you — and, crucially, what it does not.
  • Where fine-tuning, instruction-tuning, and RAG pick up, at a boundary-marking level.

Generation is the inference you already know

During training the model saw whole sequences and was scored on every position at once. To generate, you run the same model forward only, one token at a time:

  prompt tokens ─▶ forward pass ─▶ logits ─▶ sample a token ─▶ append it
        ▲                                                         │
        └───────────────── feed it back in ───────────────────────┘
                   repeat until a stop token or a length cap

That is autoregressive decoding — the exact loop from Tokens, context and generation and Prefill and decode. The weights are now fixed (no loss, no backprop, no updates); each step reads the whole model to produce one token, which is why decode is memory-bound and why the KV cache exists. Everything the serving half of this book optimised — batching, paging, quantization, speculative decoding — is optimising this loop, running these trained weights.

Key idea

Training and generation are the same forward pass pointed in opposite directions. Training runs it on full sequences to compute a loss and update weights; generation runs it one token at a time on frozen weights to produce text. This is why the book could teach inference first: you were running the trained artifact all along, and this part just showed where it came from.

What pre-training gives you — and what it doesn't

The process this part described — next-token prediction over a large corpus — is pre-training. It produces a base model: something with a deep statistical grasp of language, facts, and patterns, able to continue text plausibly. That is a lot. But a base model is not yet a helpful assistant:

  • It continues text; it does not reliably follow instructions or answer in the format you want. Ask a base model a question and it may just continue with more questions.
  • Its knowledge is frozen at the training cut-off and lives only in its weights — it cannot look anything up.
  • It has no built-in sense of which continuations are helpful, safe, or true — only which are likely given the corpus.
Watch out

"Trained" is not "aligned to your task." Pre-training optimises for likely next tokens, not correct or helpful ones — the same "likely is not correct" gap the evaluation half of this book is built around. A base model is raw capability; turning it into a product takes the stages below.

What the next stages add (the boundary this part drew)

This part deliberately stopped at pre-training. The techniques that turn a base model into a useful system are their own topics, but here is the map so you know where the edges are:

  • Fine-tuning continues training the weights on a smaller, targeted dataset to specialise the model (a domain, a style, a task). Same loop as this part — forward, loss, backprop, update — just on curated data, often with parameter-efficient methods (like LoRA) that adjust a small slice of the weights.
  • Instruction-tuning and preference alignment (SFT, RLHF/DPO) fine-tune specifically so the model follows instructions and prefers helpful, harmless answers — the step that turns a text-continuer into an assistant.
  • RAG (retrieval-augmented generation) leaves the weights alone and instead feeds relevant documents into the prompt at inference, giving the frozen model fresh or private knowledge it never trained on — see Application patterns.
  • Steering at inference (prompting, decoding controls, and more advanced activation-level methods) nudges behaviour without changing weights at all.
Key idea

Pre-training builds capability; fine-tuning and alignment build behaviour; RAG and prompting supply knowledge and control at inference. They stack: almost every deployed system is a pre-trained base, adapted by one or more of these, then served by the inference stack this book began with.

The same recipe, scaled

The small model in this part and a frontier model differ mostly in scale, not in kind: more and cleaner data, more parameters (wider, deeper), and far more compute through the same loop. Understanding the small version end to end is most of the way to understanding the large one — the extra is engineering and scale, not new principles.

Mental model

Training ends with fixed weights; generating from them is the inference loop you already knew. Pre-training buys raw capability but not helpfulness, freshness, or truth — fine-tuning and alignment shape behaviour, RAG and prompting supply knowledge and control, and the whole thing is then served by the inference stack. Same forward pass, start to finish.

Common mistakes

  • Thinking a base model is an assistant. Pre-training yields a text-continuer; following instructions comes from a later tuning stage.
  • Believing a model "knows" current facts. Its knowledge is frozen at the cut-off; fresh or private facts come from RAG, not the weights.
  • Assuming generation differs from inference. It is the identical forward pass on frozen weights — everything the serving chapters optimised applies directly.
  • Equating low loss with a good product. Likely ≠ correct ≠ helpful — that gap is why the evaluation half of this book exists.

Practical guidance

  • Pick the cheapest stage that solves your problem: prompt/RAG (no training) before fine-tuning (weights) before pre-training (rarely yours to do).
  • Reach for RAG when the gap is knowledge (fresh, private, citable), and for fine-tuning when the gap is behaviour/format/style the base model can't be prompted into reliably.
  • Remember a fine-tune is this same training loop on curated data — the concepts in this part transfer directly.
  • Whatever stage you use, the result is still weights served by the inference stack — so the serving and evaluation halves of this book remain exactly as relevant.

Summary

  • Generating from trained weights is the autoregressive inference loop you already know, run on frozen weights — training and generation are one forward pass, both directions.
  • Pre-training yields a capable base model but not a helpful, current, or truthful assistant.
  • Fine-tuning/alignment shape behaviour, RAG and prompting supply knowledge and control at inference — they stack on top of the base.
  • A frontier model is the same recipe at scale; understanding the small one is most of the understanding.

Knowledge check

You pre-train a base model and it produces fluent text but ignores instructions and invents recent "facts." Which stage fixes each problem?

Ignoring instructions is fixed by instruction-tuning / preference alignment (SFT then RLHF/DPO) — fine-tuning the weights specifically to follow instructions and prefer helpful answers. Inventing recent facts is a knowledge gap: the weights are frozen at the training cut-off, so the fix is RAG — retrieve current/authoritative documents into the prompt at inference — not more pre-training. Capability was there; behaviour and knowledge are separate additions.

Why was it possible for this book to teach inference before training?

Because generation from a trained model is inference: the same forward pass, run one token at a time on fixed weights. The inference chapters were running the trained artifact the whole time; this part simply explained where those weights came from. Training and serving meet at the frozen weight matrices — the object both halves of the book are about.

Related chapters