Part 8 — Production Observability

Prompt and response tracing

Tracing LLM systems·Serving·7 min read

The one thing traditional observability never had to log is the thing LLM debugging most depends on: the actual assembled prompt and the raw model response. This chapter is about capturing them — the exact text that went in and came out — because without it, an LLM failure is essentially uninvestigable.

Ordinary request tracing logs the API call: endpoint, status, latency. For an LLM system that's not enough, because the behaviour you're debugging is determined by two things ordinary observability never captured — the exact assembled prompt that went to the model, and the raw response that came back before your code touched it. These are the LLM-specific additions to the trace, and they're the single highest-value thing you can log, because almost every LLM failure investigation begins with "what exactly did we send, and what exactly did we get?"

What you will understand by the end

  • Why the assembled prompt (not the template) is what you must capture.
  • Why the raw model response (before parsing) is essential.
  • How prompt/response traces make otherwise-invisible failures diagnosable.
  • The privacy weight this data carries, handed to a later chapter.

Capture the assembled prompt, not the template

The prompt is assembled per request from a system prompt, retrieved context, the user input, and instructions. The template is in your code; the assembled prompt — the exact final text the model actually saw — exists only at request time and is where the bugs hide: a context that overflowed the window, a retrieval that injected the wrong document, an instruction that got buried. To debug, you need the rendered prompt, not the template you think produced it.

Key idea

Log the fully assembled prompt, exactly as the model received it. The template tells you what you meant to send; the assembled prompt tells you what you actually sent — and the gap between those two (a bad retrieval, an overflow, a mis-ordered section) is where a huge fraction of LLM failures live. You cannot reconstruct the assembled prompt after the fact; capture it at the moment.

Capture the raw response, before parsing

Equally essential is the model's raw output — the exact text it produced, before your parser and validator touched it. By the time an error surfaces downstream, the raw output may have been transformed, repaired, or discarded, erasing the evidence. The raw response is what tells you whether the model produced something malformed (a structural failure), something valid-but-wrong (a semantic failure), or something your parser mishandled — three very different bugs that look identical once the raw text is gone.

Watch out

If you only log the parsed result or the final answer, you've thrown away the evidence needed to tell a model failure from a parsing failure. A wrong final answer with no raw response logged is unattributable — you can't tell if the model said the wrong thing, said the right thing malformed, or said the right thing and your code broke it. Capture the raw output before any transformation.

Why this makes failures diagnosable

With the assembled prompt and raw response in the trace, an LLM failure investigation becomes concrete instead of speculative. "The answer was wrong" becomes: the assembled prompt shows the retrieval fetched the wrong section, so the model answered correctly from bad context — a retrieval fix. Or: the prompt was right, the raw response chose the wrong pattern — a model/prompt fix. Or: the raw response was correct but the parser dropped a field — a plumbing fix. That stage-localisation is impossible without the two pieces of text this chapter captures; with them, production debugging is just failure analysis with the evidence in hand.

Observed evidence

This project's trials preserve exactly this: each trace shows the raw model intent alongside the corrected and expected intents, so you can see whether the model produced a valid-but-wrong object (it did — the top_n-for-breakdown error) versus a parsing issue (none — everything was schema-valid). That raw-vs-processed capture is precisely what let the failure be diagnosed as semantic rather than structural. In production, logging the raw response buys the same power on live traffic. See raw model output beside the processed result →

The privacy weight

Prompts and responses are the most sensitive thing you'll log: they contain the user's actual input and the model's actual output, often including personal or confidential data. So this chapter's "log everything" instinct collides directly with privacy obligations — capture is essential for debugging and a liability if mishandled. Privacy and trace retention is where that tension gets resolved (redaction, retention limits, access control); flag it here so the capture habit and the responsibility arrive together.

Mental model

Log the exact assembled prompt (what the model actually saw, not the template) and the raw model response (before parsing). The template-vs-assembled gap and the model-vs-parser distinction are where LLM failures hide, and both are unreconstructable after the fact. This text is the highest-value thing you can trace — and the most sensitive, so it comes with a privacy obligation.

Common mistakes

  • Logging the template, not the assembled prompt. You miss the retrieval, overflow, and ordering bugs that only exist in the rendered text.
  • Logging only the parsed/final result. You can't distinguish a model failure from a parsing failure without the raw response.
  • Capturing after transformation. Repairs and parsing erase the evidence; capture the raw output first.
  • Ignoring the privacy weight. Prompt/response data is the most sensitive you log; capture responsibly from the start.

Practical guidance

  • Capture the fully assembled prompt exactly as sent, and the raw model response exactly as received, in every trace.
  • Keep them before any parsing/repair, so you can attribute a failure to model, prompt, or parser.
  • Pair the capture with a privacy plan (redaction, retention, access) from day one, since this is your most sensitive data.
  • Make prompt/response viewable in the trace UI beside expected output, the way offline failure analysis reads them.

Summary

  • The assembled prompt and raw response are the LLM-specific trace fields — the exact text in and out.
  • Log the assembled prompt (not the template) and the raw response (before parsing); otherwise the highest-value evidence is gone.
  • Together they let you localise a failure to model, prompt, or parser — production failure analysis with the evidence in hand.
  • This data is the most sensitive you log, so capture comes with a privacy obligation.

Knowledge check

A user got a wrong answer. Your trace logged the template prompt and the final parsed result, but not the assembled prompt or the raw response. Why might you be unable to diagnose it?

Because the two most diagnostic pieces are missing. Without the assembled prompt you can't see what the model actually received — a bad retrieval, a context overflow, or a buried instruction would be invisible, since the template looks fine. Without the raw response you can't tell whether the model produced a valid-but-wrong answer, a malformed one your parser repaired or dropped, or a correct one your code mangled — three different bugs that look identical once you only have the final parsed result. You'd be left guessing among model, prompt, retrieval, and parser with no evidence to localise the failure.

Why is prompt/response tracing both the highest-value and the most sensitive thing you log?

Highest-value because nearly every LLM failure investigation starts with "what exactly did we send and receive?" — the assembled prompt and raw response are what make a failure attributable to model, prompt, retrieval, or parser rather than a guess. Most sensitive because that same text is the user's real input and the model's real output, which routinely contain personal or confidential data. So the capture you most need for debugging is also the data that carries the greatest privacy and retention obligation — which is why it must be captured and governed (redaction, retention limits, access control) together.

Related chapters