Ambiguity and annotation disagreement

Not every input has one right answer. Some are genuinely ambiguous — two competent people, reading carefully, land on different correct interpretations. When that happens, annotators disagree, and the instinct is to treat the disagreement as a mistake to be stamped out. That instinct is often wrong. Disagreement is information: it tells you where the task itself is underspecified, and how you handle it decides whether your evaluation is honest or quietly arbitrary.

What you will understand by the end

• The difference between genuine ambiguity and label error.
• Why annotation disagreement is signal, and how to measure it.
• How to handle ambiguous cases in the contract instead of at scoring time.
• Why forcing a single answer onto a truly ambiguous input corrupts the number.

Genuine ambiguity versus label error

Two very different things both show up as "annotators disagree":

• Label error. The task has a single correct answer, but an annotator was careless, misread, or misunderstood the guidelines. This is noise; the fix is better guidelines, training, or adjudication.
• Genuine ambiguity. The input truly supports more than one correct reading — the question is underspecified, the phrasing is dual, the context is missing. No amount of annotator care resolves it, because the ambiguity is in the data, not the annotator.

Telling them apart is the first move. If your best, most careful annotators still split on the same items, you are looking at genuine ambiguity, not error.

Measure it — inter-annotator agreement

Because disagreement is signal, you measure it. Have multiple annotators label the same items and compute an agreement rate (how often they match, adjusted for chance). Low agreement on a slice tells you that slice is ambiguous or the taxonomy is muddy — and that any model score on it will be unstable, because even humans don't converge. Agreement is a ceiling: a system can't be reliably "more correct" than the humans defining correct can agree on.

Handle ambiguity in the contract, not at scoring time

Genuine ambiguity has to be dealt with before scoring, in the contract, not improvised case by case. The options:

• Accept multiple answers. Score any of the valid readings as correct (a small reference set instead of a single answer).
• Exclude the ambiguous cases. Remove them from the scored set so they don't add noise — documented, not silently dropped.
• Disambiguate the input. Add the missing context so it has one answer, turning an ambiguous case into a clear one.
• Pick a canonical reading and state it in the contract, so scoring is consistent even if the choice is somewhat arbitrary.

The one unacceptable option is deciding at scoring time, per case, which resurrects the fuzzy contract and makes the number depend on the grader.

Mental model

Some inputs genuinely have more than one right answer; honest annotators disagree on them. Distinguish that from label error, measure it as inter-annotator agreement (a ceiling on your score), and decide how to handle ambiguous cases in the contract — accept multiple, exclude, disambiguate, or canonicalise — never case by case at scoring time.

Common mistakes

• Treating all disagreement as annotator error. Genuine ambiguity is in the data; more guidelines won't fix it.
• Ignoring inter-annotator agreement. Without it you can't tell a well-specified slice from a contested one, and you'll over-trust scores on ambiguous data.
• Resolving ambiguity at scoring time. Case-by-case decisions make the number depend on the grader — a fuzzy contract by the back door.
• Reporting a confident number on a low-agreement slice. The ground truth isn't settled; neither is the score.

Practical guidance

• Double-label a sample and compute inter-annotator agreement; low-agreement slices are ambiguous or have taxonomy problems.
• Decide the ambiguity policy in the contract — accept-multiple, exclude, disambiguate, or canonicalise — before scoring, and apply it uniformly.
• Prefer disambiguating the input or designing single-answer tasks where you can; it makes everything downstream stable.
• Read human disagreement as a ceiling on the score, and don't over-interpret model differences on contested slices.

Summary

• Disagreement is either label error (noise, fixable) or genuine ambiguity (in the data, not the annotator) — distinguish them first.
• Annotation disagreement is signal: measure it as inter-annotator agreement, which caps the meaningful score.
• Handle ambiguous cases in the contract (accept-multiple / exclude / disambiguate / canonicalise), never per case at scoring time.
• This project designs ambiguity out up front, which is a big reason its numbers are stable.

Knowledge check

On one slice of your data, two expert annotators agree only 68% of the time, and your model scores 84% there. What should you conclude?

The 84% is not trustworthy: you're scoring against labels the experts themselves contest 32% of the time, so the "ground truth" on that slice isn't settled and the model score is measuring partly-arbitrary references. The slice is genuinely ambiguous (or the taxonomy is unclear). Fix it upstream — disambiguate the inputs, accept multiple valid answers, or exclude the slice — and don't report a confident number there until human agreement is high enough to define correct.

Why must ambiguous cases be handled in the contract rather than decided during scoring?

Because deciding per case at scoring time reintroduces grader dependence — the same output earns different scores depending on who grades it or on what day — which is exactly the fuzzy-contract failure that makes numbers unreproducible. Setting the policy in the contract up front (accept multiple, exclude, disambiguate, or pick a canonical reading) applies one consistent rule to every ambiguous case, keeping the score a property of the system, not the grader.

Related chapters

• The evaluation contract — where the ambiguity policy must be decided
• Taxonomy and label design — disagreement that's really a category-design bug
• Ground truth and reference answers — single vs multiple valid references
• Sample size and uncertainty — the other reason a score can be less certain than it looks