Part 8 — Production Observability

User feedback

Measuring production quality·Evaluation·6 min read

Users are constantly telling you whether an answer was good — mostly without clicking anything. This chapter is about collecting both explicit feedback (thumbs, ratings, reports) and the far richer implicit feedback (edits, retries, abandonment), understanding each one's biases, and turning it into a quality signal you can trust.

The most direct read on production quality comes from the people using the system. But most user feedback is not the thumbs-up button — it's what users do: accept an answer, rewrite it, ask again, or give up. Learning to collect both the explicit feedback users deliberately give and the implicit feedback their behaviour leaks — and to read each through its biases — turns a stream of user actions into one of your most valuable quality signals.

What you will understand by the end

  • The difference between explicit and implicit user feedback.
  • Why implicit feedback is more abundant and often more honest.
  • The biases in each — vocal minority, selection, and behavioural confounds.
  • How user feedback feeds the evaluation loop.

Explicit versus implicit

  • Explicit feedback is what users deliberately provide: thumbs up/down, star ratings, "report a problem," free-text comments. It's clean and interpretable — a thumbs-down is unambiguous — but rare: only a small, self-selected fraction of users ever clicks anything, so it's sparse and skewed.
  • Implicit feedback is what their behaviour reveals without any deliberate action: do they accept the answer as-is, edit it before using it, retry/regenerate, abandon the session, or complete the downstream task? It's abundant (every request generates it) and often more honest — behaviour is harder to fake than a rating — but indirect and confounded.
Key idea

Most of your quality signal is implicit. Explicit feedback is clean but comes from a tiny, unrepresentative slice; implicit feedback comes from everyone and reflects what they actually did. A user who silently rewrites half your answer told you it was mediocre more reliably than the one-in-a-thousand who clicked thumbs-down. Instrument for behaviour, not just buttons.

The biases in each

Neither kind is a clean measurement, and using them well means knowing how they lie:

  • Explicit — vocal minority & selection bias. The users who leave feedback are unrepresentative: disproportionately the very angry or very pleased, not the typical user. A ratings average is a measurement of raters, not users.
  • Implicit — behavioural confounds. An edit might mean the answer was wrong — or that the user wanted a different tone. An abandonment might mean failure — or a phone call. A retry might mean a bad answer — or curiosity. The behaviour correlates with quality but is caused by other things too.
Watch out

A high thumbs-up rate is not proof of quality — it's proof that the small minority who click thumbs are happy, which can coexist with widespread quiet dissatisfaction. And a single implicit signal (edits) has confounds. Treat user feedback as evidence to triangulate, corrected for its selection and behavioural biases, not as a direct quality score. Confirm suspected problems with sampled review.

From feedback to the evaluation loop

User feedback's highest value isn't the dashboard number — it's as a source of real failing cases. A thumbs-down or a heavily-edited answer points at a specific input the system got wrong, which is exactly the raw material the incidents-to-tests loop runs on: surface the failure, root-cause it, capture it as a test case, add it to the offline suite. Explicit reports are especially good for this — a user who bothered to report a problem has handed you a labelled failure. User feedback is how production tells your evaluation set what to test next.

Observed evidence

This project's ground truth was authored deliberately rather than gathered from users — a fixed set of expected intents — which is the offline luxury production trades away. A live version would lean heavily on user feedback: edits to the generated query, users rephrasing when the result looked wrong, explicit "this isn't what I asked" reports — each an implicit or explicit signal to mine for the failing questions that become the next hidden-test cases. The offline set is hand-built; production sets are grown from feedback. The kind of failing case user feedback would surface →

Mental model

Users constantly signal quality — mostly implicitly (accept, edit, retry, abandon, task success), occasionally explicitly (thumbs, ratings, reports). Implicit is abundant and honest but confounded; explicit is clean but a vocal, self-selected minority. Read each through its biases, triangulate, and mine both for the real failing cases that feed the offline suite.

Common mistakes

  • Relying only on explicit feedback. It's sparse and from a vocal minority; you miss the silent majority's experience.
  • Reading a ratings average as quality. It measures raters, not users — selection bias baked in.
  • Taking one implicit signal at face value. Edits, retries, and abandonment all have non-quality causes; triangulate.
  • Collecting feedback but not feeding it back. Its best use is surfacing failing cases for the offline suite, not a vanity number.

Practical guidance

  • Instrument implicit feedback (accept/edit/retry/abandon, downstream success) as first-class signals — it's most of the signal and comes from everyone.
  • Offer low-friction explicit feedback too, but read it as the vocal minority it is, corrected for selection bias.
  • Triangulate feedback with sampled scoring and reliability rates before trusting a quality story, and mind each signal's confounds.
  • Route thumbs-downs, reports, and heavily-edited answers into the incidents-to-tests loop as likely failing cases.

Summary

  • User feedback is explicit (thumbs, ratings, reports — clean but a vocal minority) and implicit (edits, retries, abandonment, task success — abundant, honest, confounded).
  • Most of the signal is implicit; instrument behaviour, not just buttons.
  • Each kind has biases — vocal-minority/selection for explicit, behavioural confounds for implicit — so triangulate rather than trust one.
  • Its highest value is as a source of real failing cases feeding the offline suite via the incidents-to-tests loop.

Knowledge check

Your product's thumbs-up rate is 95%, but a sampled review finds many mediocre answers. How do you reconcile these, and which do you trust?

They're consistent once you account for selection bias: only a tiny, self-selected minority clicks thumbs at all, disproportionately the very satisfied (and very angry), so a 95% thumbs-up rate measures those raters, not the typical user — it can easily coexist with widespread quiet mediocrity that never generates an explicit rating. Trust the sampled review, which scores a representative slice rather than a self-selected one, and read the implicit signals (edit and retry rates) to corroborate. The thumbs rate isn't wrong, it's just answering a narrower question than "is the system good for users?"

Users frequently edit the system's answers before using them. Why is that a strong quality signal but not conclusive on its own?

Strong because it's implicit behaviour from everyone — a user bothering to rewrite an answer is costly, honest evidence that the answer wasn't good enough as delivered, and it comes from the whole population, not a vocal minority. Not conclusive because editing has confounds: some edits are about correctness, but others are tone, formatting, personalization, or house style — none a quality defect. So a high edit rate flags a likely quality issue but needs triangulation (do retries and sampled scores agree?) and a look at what users are changing before you conclude the answers are wrong rather than merely not in the user's preferred style.

Related chapters