Part 8 — Production Observability

Dashboards, alerts and incident investigation

Operating the system·Serving·7 min read

Part 8 closes by putting the pieces to work: dashboards that surface the signals worth watching, alerts that page you when they move, and the investigation workflow that turns an alert into a root cause and a fix — the moment observability pays off, and the on-ramp back into the continuous-evaluation loop.

You've instrumented traces, captured prompts and responses, tracked metrics, gathered quality signals, watched for drift, and governed the data. This final chapter of Part 8 is where all of it does its job: dashboards that show the right signals at a glance, alerts that summon you when something moves, and the investigation workflow that walks from "something's wrong" to "here's the cause and the fix." This is the moment observability earns its keep — and the point where production hands back to the continuous-evaluation loop.

What you will understand by the end

  • What belongs on an LLM system's dashboards — operational and quality signals.
  • How to design alerts that page on real problems without crying wolf.
  • The investigation workflow from alert to root cause, using the traces you captured.
  • How incident investigation closes the loop back into evaluation.

Dashboards: operational and quality, side by side

A dashboard is a curated view of the signals worth watching continuously. For an LLM system it must show both layers, because either alone lies:

The monitoring-vs-observability lesson made concrete: a dashboard of green operational metrics with no quality panel will show "healthy" through a semantic collapse. Put quality signals next to the operational ones so no one reads uptime as correctness.

Key idea

An LLM dashboard must pair operational health with quality signals, because a system can be fast, cheap, and up while its answers are wrong. Show both layers together, gated on percentiles and trends, so the dashboard can surface the semantic failures that operational metrics structurally cannot.

Alerts: page on real problems, not noise

An alert is a promise to interrupt someone. Design them so that promise is worth keeping:

  • Alert on percentile and trend movements (p99 latency breach, a drift trend, a fallback-rate spike), not on single noisy data points.
  • Tune thresholds against uncertainty so normal variation doesn't page — alert fatigue from crying wolf is how real alerts get ignored.
  • Alert on the reliability-layer rates (retry/fallback/guardrail) and quality proxies, not just operational metrics — those often move first when quality degrades.
Watch out

Two failure modes bracket good alerting. Too sensitive and every noise blip pages someone until the team mutes the channel — and then misses the real incident. Too coarse (or quality-blind) and a semantic degradation never fires an alert at all, because you only watched operational metrics. Alert on the right signals (including quality and reliability rates) at noise-aware thresholds — precision matters as much as coverage.

Investigation: from alert to root cause

When an alert fires (or a quality signal degrades), observability pays off — because everything this part captured is now the evidence. The workflow is the production form of root-cause analysis:

   ALERT / signal ─▶ SCOPE ─▶ OPEN TRACES ─▶ LOCALISE ─▶ ROOT CAUSE ─▶ FIX ─▶ CAPTURE
   what moved?        which     the failing    which      why           deploy   → test case
                      requests? cases           stage?    (drift? model  (gated)  (incidents-
                      (filter)  (prompt+resp,             change? new             to-tests)
                                retrieval/tool)           input type?)
  • Scope — what moved, for whom, since when (filter the dashboards).
  • Open the traces — read the actual failing requests: assembled prompt, raw response, retrieval/tool steps.
  • Localise — which stage failed (the pipeline view).
  • Root cause — why: a drift, a hosted-model change, a new input type, a bad deploy.
  • Fix — deploy the fix through the gate.
  • Capture — turn the incident into a test case so it can't silently recur.

Closing the loop

That last step is the whole point, and where Part 8 rejoins Part 9. An incident isn't done when the alert clears — it's done when the failure is a permanent test case in the offline suite. Production observability is how the continuous-evaluation loop detects what reality breaks; the loop is how it stops that failure from returning. Observe → investigate → root-cause → capture → gate → and the offline evaluation is now stronger than before the incident.

Observed evidence

This project is the investigation surface, run offline: the dashboard pairs a quality view (per-system accuracy) with openable traces, so you can go from "the 3B system is low" to opening its failing trials to localising the ranking-vs-breakdown cause — the exact scope → open → localise → root-cause path this chapter describes, just on a fixed dataset instead of a live alert. Point that same workflow at production traffic and alerts, and you have Part 8 end to end. The dashboard-plus-traces investigation surface →

Mental model

Dashboards pair operational health with quality signals (green ops isn't green quality); alerts fire on percentile/trend moves of the right signals at noise-aware thresholds (not blips, not quality-blind); investigation walks alert → scope → open traces → localise → root cause → fix → capture. The last step returns the incident to the offline suite as a test — observability detects what reality breaks, and the continuous loop stops it recurring.

Common mistakes

  • Operational-only dashboards. Green metrics hide semantic failures; show quality signals beside them.
  • Noisy alerts. Paging on blips breeds alert fatigue until real alerts are muted; tune to noise-aware thresholds.
  • Quality-blind alerting. If nothing alerts on quality proxies or reliability rates, a semantic degradation goes unnoticed.
  • Closing an incident at "alert cleared." It's not done until the failure is a captured test case guarding against recurrence.

Practical guidance

  • Build dashboards that show operational + quality signals together, gated on percentiles and trends.
  • Alert on the right signals (tail latency, drift trends, reliability/quality proxies) at noise-aware thresholds; protect against alert fatigue.
  • Run a standard investigation workflow — scope, open traces, localise, root-cause, fix, capture — leaning on everything Part 8 instrumented.
  • Close every incident into a test case via the incidents-to-tests loop, so the offline suite grows stronger with each one.

Summary

  • Dashboards must pair operational health with quality signals — green ops can hide a semantic collapse.
  • Alerts should fire on percentile/trend moves of the right signals at noise-aware thresholds, avoiding both fatigue and quality blindness.
  • Investigation walks alert → scope → open traces → localise → root cause → fix → capture, the production form of root-cause analysis.
  • The loop closes when the incident becomes a permanent test case — observability detects what reality breaks; continuous evaluation stops it recurring.

Knowledge check

An alert fires: p99 latency and the fallback rate both jumped. Walk the investigation from here to a closed loop.

Scope: check the dashboards — which requests, which segment, since when; the paired jump suggests the primary path is failing and routing to fallback (raising both fallback rate and, via retries/fallback, tail latency). Open traces for the affected requests — assembled prompt, raw response, retrieval/tool steps — to see what's actually failing. Localise the stage (e.g. the model erroring, or a retrieval timeout triggering fallback). Root-cause it — a hosted-model change, an input-drift spike, a bad deploy, an upstream dependency. Fix and ship through the gate. Capture: turn the failing cases into permanent test cases in the offline suite via incidents-to-tests, so the same failure can't silently recur. The incident closes not when latency recovers but when the regression test exists.

Why must an LLM dashboard show quality signals next to operational metrics, and why is alerting only on operational metrics insufficient?

Because operational metrics (latency, cost, uptime, validity) measure whether the system runs, not whether it's right — and LLM systems fail semantically, staying fast, cheap, up, and valid while producing wrong answers. A dashboard or alerting scheme that watches only operational signals will show "healthy" and page no one straight through a quality collapse, because no operational metric moves. Pairing quality proxies (edit/retry rates, sampled scores, drift trends) with the operational ones — and alerting on them — is what lets the system surface the semantic degradations that are its most important and most invisible failures.

Related chapters