Part 3 — Evaluation Foundations

Exact match, semantic scoring and rubric scoring

Scoring methods·Evaluation·6 min read

Turn the deterministic-vs-subjective idea into concrete methods. Exact match, semantic scoring, and rubric scoring sit on a spectrum from strict-and-cheap to flexible-and-costly — each with a signature failure mode. Matching the method to the task is where evaluations succeed or quietly lie.

You've decided what "correct" means and where your ground truth comes from. Now you need an actual method to turn an output into a score. Three cover most of the field — exact match, semantic scoring, and rubric scoring — and they line up on a spectrum from strict, cheap, and brittle to flexible, expensive, and fuzzy. Each has a signature failure mode, and picking the wrong one is how an evaluation produces confident, wrong numbers.

What you will understand by the end

  • What exact match, semantic scoring, and rubric scoring each do.
  • The signature failure mode of each — false negatives, false positives, judge dependence.
  • How to match the method to the task's ground-truth shape.
  • Why this project can legitimately use the strictest method.

The spectrum

   strict / cheap ◄──────────────────────────────────────► flexible / costly
   EXACT MATCH              SEMANTIC SCORING              RUBRIC SCORING
   equal to reference?      close enough in meaning?      meets each criterion?
   binary, mechanical       similarity threshold          checklist, often LLM-judged
   fails on paraphrase      passes plausible-but-wrong     depends on judge quality

Exact match

Is the output equal to the reference (string equality, or structural equality of a normalized object)? Cheap, fully deterministic, unambiguous. Its signature failure is the false negative: an answer that is correct but worded or ordered differently gets marked wrong. It only works when the task has a canonical answer — or when you canonicalize outputs before comparing (sort fields, normalize case) so harmless variation doesn't count as error.

Semantic scoring

Is the output close enough in meaning to the reference — via embedding similarity, normalized overlap, or a learned metric — passing above a threshold? Tolerant of surface variation, so it rescues the paraphrases exact match rejects. Its signature failure is the false positive: a fluent answer that is near the reference in embedding space but says the wrong thing scores as correct. Semantic similarity is not truth.

Rubric scoring

Break "good" into a checklist of criteria — is it faithful to the source, does it answer the question, is the format right — and score each, usually with a human or an LLM judge. The only workable method for open-ended outputs with many valid forms. Its signature dependence is the judge: the score is only as good as the rubric's clarity and the judge's calibration.

Key idea

Each method fails in a characteristic direction: exact match under-credits (false negatives on equivalent answers), semantic scoring over-credits (false positives on plausible-but-wrong), and rubric scoring inherits its judge's variance. Choosing a method means choosing which error you can live with for this task.

Match the method to the task

The single-vs-many shape of your ground truth mostly decides this:

  • One canonical answer (labels, structured intents, computed values) → exact match on a normalized form. Strictest, cheapest, most reproducible — use it whenever you can.
  • Equivalent answers vary on the surface but there's still a "right meaning" → semantic scoring, with the false-positive risk in mind (and often a human spot-check).
  • Many genuinely valid answers (summaries, explanations, code behaviour) → rubric scoring, decomposed into checkable criteria.
Watch out

The seductive failure is semantic scoring's false positive on structured tasks: an output that's embedding-close to the reference but selects the wrong metric or filter passes. For anything with a canonical answer, prefer exact match on a normalized form — it cannot be fooled by plausibility. Reach for semantic scoring only when surface variation is real and unavoidable.

Why this project uses the strictest method

Observed evidence

This project scores with exact match against the expected QueryIntent — the strictest method — and it's the right call precisely because the task was designed to have a canonical answer. The intent is normalized, so harmless ordering doesn't cause false negatives, and exact match refuses to be fooled by a plausible-but-wrong intent (the "top-N vs breakdown" error). A semantic scorer might have passed that wrong-but-similar intent; exact match caught it. See exact-match failures →

Mental model

Three methods on a strict→flexible spectrum: exact match (equal? — cheap, false negatives), semantic (close enough? — tolerant, false positives), rubric (meets criteria? — open-ended, judge-dependent). Pick by your ground truth's shape, and choose the failure direction you can tolerate. When a canonical answer exists, exact match on a normalized form wins.

Common mistakes

  • Exact match without normalization. Penalizes equivalent answers (reordered fields, different casing) as wrong — false negatives that understate the system.
  • Semantic scoring on structured tasks. Its false positives pass plausible-but-wrong outputs; exact match on a normalized form is safer.
  • Rubric scoring without a clear rubric. Vague criteria make the judge the real variable; the score becomes noise.
  • One method for every task. The right method depends on whether the answer is canonical, surface-variant, or genuinely plural.

Practical guidance

  • Prefer exact match on a normalized form wherever a canonical answer exists — normalize away harmless variation first so you don't manufacture false negatives.
  • Use semantic scoring only when surface variation is unavoidable, and spot-check its passes for plausible-but-wrong false positives.
  • For open-ended tasks, write an explicit rubric of checkable criteria and calibrate the judge (Part 9).
  • Report the method with the number; "94% exact-match" and "94% semantic" are different claims.

Summary

  • Exact match (strict, cheap, false negatives), semantic scoring (tolerant, false positives), and rubric scoring (open-ended, judge-dependent) span the scoring spectrum.
  • Each has a signature error; choosing a method is choosing which error you can tolerate.
  • Match the method to the ground truth: canonical → exact match; surface-variant → semantic; plural → rubric.
  • This project's normalized exact match is the right, strictest choice — it caught a plausible-but-wrong intent a semantic scorer might have passed.

Knowledge check

Your extraction task outputs a structured record with a canonical correct value per field, but exact match reports 60% while spot-checks look much better. What's the likely bug, and the fix?

Exact match is producing false negatives from un-normalized harmless variation — field ordering, casing, whitespace, equivalent formats — so correct records are scored wrong. Fix: normalize/canonicalize both the output and the reference before comparing (sort fields, standardize formats), then exact match will credit equivalent-but-differently-written records. The task still has a canonical answer, so keep exact match — just compare normalized forms.

Why is semantic (embedding-similarity) scoring risky for a query-intent task?

Because its signature failure is the false positive: an intent that is embedding-close to the reference but selects the wrong metric, filter, or aggregation would score as correct even though it produces a different, wrong query. For a task with a canonical answer, exact match on a normalized intent can't be fooled by plausibility, so it's the safer method; semantic similarity is not the same as correctness.

Related chapters