Deterministic and subjective scoring

There are only two fundamental kinds of scoring. Either a rule decides right from wrong the same way every time, or a judgment is required and the score depends on who's judging. Deterministic scoring is cheap, fast, and perfectly reproducible; subjective scoring is expensive, slow, and variable. Most of the craft of evaluation is arranging your task so that as much of it as possible can be scored deterministically — and knowing, honestly, which parts can't.

What you will understand by the end

• The difference between deterministic and subjective scoring.
• Why deterministic scoring is the default to reach for, and its blind spots.
• Why subjective scoring is sometimes unavoidable, and what it costs.
• How the task's output shape decides which kind you get.

The two kinds

• Deterministic (objective) scoring. A rule maps an output to a score with no judgment: does it equal the reference, does it parse, does it execute, does the number match. Run it a thousand times and get the same answer; run it on another machine and get the same answer. It is cheap, instant, and reproducible.
• Subjective scoring. A person (or a model standing in for one) must decide: is this summary faithful, is this answer helpful, is this tone appropriate. There's no mechanical rule, so the score carries the judge's variance — two judges, or one judge twice, can differ.

   DETERMINISTIC                     SUBJECTIVE
   a rule scores it                  a judge scores it
   same answer every time            varies by judge / mood
   cheap · instant · reproducible    costly · slow · variable
   exact match, parses?, executes?   helpful? faithful? well-written?

Push toward deterministic — but know its blind spot

Whenever the task admits a single canonical answer (ground truth), deterministic scoring is available and you should use it: classification labels, structured intents, computed results. It gates every change without human cost.

But determinism has a blind spot: a rule only measures exactly what it checks. Exact match against a reference is deterministic and can be too strict (marking an equivalent answer wrong) or too shallow (passing a structurally-valid but meaningless output). Deterministic does not mean correct-by-magic; it means consistent. A well-chosen rule is consistent and meaningful; a badly chosen one is consistently misleading.

When subjective is unavoidable

For open-ended outputs with many valid answers — summaries, explanations, free-form responses — there is no rule that captures "good," so judgment is required. That's legitimate; the mistake is pretending otherwise. When you must score subjectively:

• Expect variance and manage it with written guidelines and multiple raters.
• Consider an LLM-as-judge to scale the judgment (with its own calibration caveats — Part 9).
• Turn as much as you can into checkable sub-criteria (a rubric), converting one big subjective call into several smaller, more consistent ones.

Mental model

Two kinds of scoring: a rule (deterministic — same score every time, cheap, reproducible) or a judge (subjective — varies, costly). Reach for deterministic whenever the task has a canonical answer, but make sure the rule encodes meaning, not just surface. Accept subjective only where judgment is genuinely required, and manage its variance.

Common mistakes

• Scoring subjectively when a rule would do. Wastes human effort and adds variance to something you could have measured mechanically.
• Assuming deterministic = correct. A rule is only as meaningful as what it checks; a consistent but shallow rule misleads consistently.
• Pretending an open-ended task is objective. Forcing exact match on many-valid-answer outputs marks correct answers wrong.
• Ignoring subjective-score variance. Without guidelines and multiple raters, a subjective number isn't reproducible enough to compare with.

Practical guidance

• Default to deterministic scoring wherever the task has a canonical answer; it's the only kind you can run on every change and fully trust.
• Choose the deterministic rule to encode the contract's meaning, and sanity-check it for too-strict / too-shallow behaviour.
• When judgment is unavoidable, decompose it into a rubric of checkable criteria, use guidelines and multiple raters, and consider an LLM judge to scale.
• Label each metric by kind — a deterministic exact-match and a subjective helpfulness rating are not comparable numbers.

Summary

• Scoring is either deterministic (a rule, reproducible and cheap) or subjective (a judge, variable and costly).
• Prefer deterministic whenever a canonical answer exists — but ensure the rule measures meaning, not just a convenient surface.
• Some open-ended tasks require judgment; manage its variance with rubrics, guidelines, and multiple raters (or an LLM judge).
• This project's exact-match scoring is fully deterministic, which is why its numbers carry no grader variance.

Knowledge check

Your task is multi-label classification with a fixed label set. A teammate proposes human raters score each output for "quality." What's wrong, and what's better?

Multi-label classification against a fixed set has a canonical answer per input, so it admits deterministic scoring (compare predicted labels to the reference set — e.g. per-label precision/recall). Human "quality" rating adds cost and variance to something a rule can score exactly and reproducibly. Reserve human judgment for genuinely open-ended aspects; score the label prediction mechanically.

Deterministic scoring is reproducible. Why isn't that enough on its own?

Because reproducible isn't the same as meaningful. A rule only measures what it checks, so a deterministic rule can be consistently misleading — e.g. a schema-validity check reproducibly returns 100% while correctness is 45%. You need a deterministic rule that actually encodes the contract's definition of correct, not just a convenient surface property that happens to be easy to compute.

Related chapters

• Ground truth and reference answers — the single-vs-many split that decides which kind you get
• Exact match, semantic scoring and rubric scoring — concrete methods along the spectrum
• Why valid JSON is not semantic correctness — the deterministic rule that measures the wrong thing
• LLM-as-judge — scaling subjective judgment with a model