Retrieval, tools and agents

A model on its own has two hard limits: its knowledge is frozen in its weights, and its only action is to produce text. Almost every real LLM system exists to push past those limits, and there are three ways to do it — retrieval (give it fresh knowledge), tools (let it take actions), and agents (let it work in multiple steps). Each is a way of adding components around the model, and — the theme of this book — each new component is new capability and a new failure mode you now have to evaluate.

What you will understand by the end

• The three ways to extend a model: retrieval, tools, agents — and what limit each removes.
• The components each one adds to the system, and where each can fail.
• Why extending the model always adds something new to evaluate.
• How these compose into the systems the rest of the book keeps referring to.

Retrieval: knowledge the weights don't have

The model's knowledge is frozen at its training cut-off and holds nothing private. Retrieval (RAG) fixes this by fetching relevant documents at request time and putting them in the prompt, so the model answers grounded in fetched context instead of its weights alone. It adds real components: an index, a retriever (often embedding-based), sometimes a reranker.

• What it removes: stale and missing knowledge.
• Where it fails: retrieval quality (fetching the wrong or too little context) and grounding (the model answering beyond the sources).
• What it adds to evaluate: retrieval relevance and answer groundedness — two separate measurements, per evaluating pipelines.

Tools: actions beyond emitting text

A model can only produce tokens; it can't search, compute, or call an API. Tool / function calling lets the model emit a structured call — a tool name plus arguments — which your code executes, feeding the result back. Now the system can do things: look up a record, run a calculation, hit a service. It adds a tool registry, the execution layer, and the wiring that turns a model's requested call into a real one.

• What it removes: the text-only ceiling.
• Where it fails: choosing the wrong tool, or the right tool with wrong arguments.
• What it adds to evaluate: tool-choice accuracy, argument correctness, and end-to-end task success.

Agents: multiple steps toward a goal

Retrieval and tools are single steps. An agent loops: decide → act → observe → repeat, using tools over multiple turns to complete a task no single call could. It adds a control loop, a planner, memory of what's happened, and a stopping condition.

• What it removes: the single-shot limit — tasks that need many dependent steps.
• Where it fails: compounding errors (a mistake early derails everything after) and not knowing when to stop.
• What it adds to evaluate: end-to-end task completion and step-level traces — a single final score hides where a multi-step run went wrong.

They compose — and the surface grows with them

Real systems stack these: an agent that uses retrieval as one tool among several, with structured outputs at each step. The capability is enormous, and so is the evaluation surface — every added component is another stage that can fail and another thing to measure. This is why agents especially demand step-level traces: the more steps and components, the more a single end-to-end number hides.

Mental model

A bare model is frozen knowledge that only emits text. Retrieval adds knowledge, tools add actions, agents add multi-step work — each removing a limit by adding components, and each adding its own failure modes and its own things to evaluate. They compose into powerful systems whose evaluation surface grows with every part. Add the least that solves the task.

Common mistakes

• Reaching for an agent by default. Multi-step loops add compounding failure and cost; use them only when the task needs many dependent steps.
• Treating RAG as "just prompting." Retrieval quality is half the system and needs its own metric alongside groundedness.
• Scoring an agent with one final number. It hides which step failed; agents need step-level traces.
• Forgetting each extension adds evaluation surface. New capability is new failure modes you now own.

Practical guidance

• Match the extension to the gap: retrieval for missing/stale knowledge, tools for actions, agents for genuinely multi-step tasks — and add the least that works.
• For retrieval, measure relevance and groundedness separately; for tools, tool-choice and argument correctness; for agents, end-to-end plus step-level.
• Keep as much deterministic as possible around the model (query building, execution); use the model only for the language-shaped step.
• Prefer the simplest architecture that solves the task — it's usually the most reliable and the most evaluable.

Summary

• Retrieval (knowledge), tools (actions), and agents (multi-step) are the three ways to extend a bare model.
• Each removes a limit by adding components — and adds its own failure modes and metrics.
• They compose, and the evaluation surface grows with every part; agents especially need step-level traces.
• Add the least that solves the task — this project's single-step design is why it's so cleanly evaluable.

Knowledge check

A RAG system answers a question with fluent, confident, but wrong information that isn't in any retrieved document. Which component failed, and which metric would have caught it?

This is a grounding failure (the generator answered beyond its sources), possibly compounded by a retrieval failure (the right document wasn't fetched, so the model filled the gap from its weights). Retrieval relevance and answer groundedness are separate measurements: retrieval relevance would show whether the supporting document was even fetched, and a groundedness/faithfulness metric would flag that the answer isn't supported by the retrieved context. A single end-to-end "correct?" score conflates the two; RAG needs both, which is why "just prompting" framing misses half the system.

Your task is a single question → single structured answer. A teammate proposes building it as a multi-step agent "to be safe." What's the argument against?

An agent adds a control loop, planning, memory, and multiple tool-calling steps — and with them compounding errors (an early mistake derails the rest), higher cost and latency, and a much larger evaluation surface (step-level traces, not just a final score). If the task is a single structured-output step, none of that is needed and all of it is downside. The simplest architecture that solves the task — one well-structured call, deterministically post-processed — is more reliable, cheaper, and far easier to evaluate. Reach for an agent only when the task genuinely needs many dependent steps.

Related chapters

• Application patterns — the pattern taxonomy these extensions implement
• Agent harnesses and context engineering — building the system around an agent
• Evaluating complete pipelines — measuring the stages each extension adds
• Guardrails, retries, fallbacks and correction policies — making these extended systems reliable