---
title: Annotation Disagreement: Diagnose Before Adjudicating
url: https://ypai.ai/blog/data-engineering/annotation-disagreement-adjudication-decision-table/
category: Data Engineering
published: 2026-09-06T00:00:00.000Z
modified: 2026-09-06T00:00:00.000Z
author: YPAI Engineering
tags: [Data Annotation, Quality Assurance, Inter-Annotator Agreement, Adjudication, Training Data]
---

# Annotation Disagreement: Diagnose Before Adjudicating

> Two annotators, two boxes, one partly hidden object. Which becomes ground truth is the second question. The first is why they differ.

An image contains a pedestrian standing behind a parked car. Head, shoulders and upper torso are visible. Most of the lower body is hidden. One annotator draws a tight box around the visible pixels. Another draws a taller box estimating where the person's full extent must be. The class is not in question. Both annotators are competent. The boxes differ by a lot.

Which box becomes ground truth is the second question. The first one is why they differ, because the honest answers lead to completely different actions: correct one record, rewrite one line of the specification, or change the label schema and revisit everything already annotated under the old reading. This walkthrough separates those cases and gives the table that maps each to its evidence and its consequences. It is written for data and evaluation leads who receive adjudicated labels and have to know what the adjudication actually decided.

**The scenario above is constructed for this article. It is not a YPAI project or a customer result.**

## The occluded pedestrian

An overlap measure can tell the team how far the two boxes differ. It cannot tell them why. Work presented at IEEE ICIP in 2022 makes the narrower version of this point empirically: across four user studies on detector localization, humans did not judge localization quality the way IoU scores it, so the same IoU need not correspond to the same human judgment. That study is about detector evaluation preferences rather than professional annotation agreement, and it is not an occlusion study, but it establishes that geometric overlap does not uniquely encode what people mean by a good box.

So the first diagnostic is not a metric. It is the annotation specification, and there are three readings of it.

**If the specification says "box the visible extent only."** The second annotator has made a correctable execution error against a clear rule. Preserve the original box in the QA trail, replace or reject it according to the workflow, record the error category, and then check whether the same error clusters among other occluded objects. The last step is what turns a correction into information.

**If the specification says "estimate the full physical extent."** Now the first annotator is the one inconsistent with the rule. A second question survives that finding: does the protocol say what to do when the hidden boundary cannot be estimated reliably? If it does not, the record has exposed a secondary gap, and closing only the first one leaves the second live.

**If the specification says only "draw a box around each person."** Neither annotation is wrong. The missing rule is the box semantics, and no adjudicator can supply it from the image. The correct action is to choose the convention according to the dataset's intended use, add worked examples covering partial occlusion, and determine which previously labeled occluded instances may have been affected. Picking A or B for this one record would produce a final label and leave the inconsistency alive in every similar item already delivered.

There is a fourth possibility worth naming, because teams reach for it too early. The downstream task may genuinely need uncertainty about invisible extent, in which case the right representation could be a visible box plus occlusion attributes, a difficult or ignore flag, an acceptable range, or separate visible and amodal annotations. Whether any of those is appropriate is a task-design question. Disagreement on its own is not proof that a richer representation is required.

## Six kinds of disagreement

The table below is the working version of that diagnosis. The column most teams skip is the fourth one, because each of those shortcuts produces a clean-looking dataset and leaves the cause in place.

| Disagreement type | Diagnostic evidence | Suitable response | Unsuitable shortcut | Implications for already-labeled data |
|---|---|---|---|---|
| **Execution mistake** | A clear rule exists, one annotation demonstrably violates it, and recurrence can be measured | Correct or reject the record, code the error class, inspect for recurrence | Rewriting the ontology because of an isolated slip | Target similar records if the evidence suggests a systematic pattern, otherwise local correction may suffice |
| **Unclear instruction or convention** | Competent annotators systematically choose different readings, such as visible against amodal box | Decide the convention, add worked examples, recalibrate the team | Majority vote while leaving the wording unchanged | Identify every item the ambiguous rule could have applied to and assess backfill |
| **Ontology problem** | Overlapping, missing or indistinguishable categories, with confusion persisting across annotators and slices | Revise the schema and definitions, version the change, remap or re-annotate | Blaming annotators for choosing between non-exclusive classes | Potentially broad, because the meaning of the labels has changed |
| **Instance ambiguity or insufficient evidence** | Multiple interpretations stay plausible even under a clear specification | Add evidence where it exists, otherwise preserve uncertainty, ignore flags or acceptable alternatives where the task permits | Forcing an arbitrary answer solely to raise the agreement score | Similar edge cases may need ambiguity flags or targeted review |
| **Specialist judgment** | The decision depends on knowledge general annotators do not have, and specialists can articulate the external criteria | Specialist review and adjudication with documented rationale | Treating the expert's title as proof of correctness | Review the slices needing the same specialist knowledge, since expert disagreement can itself expose a specification issue |
| **Genuine subjective variation** | Stable disagreement persists across capable raters and samples, and reflects the construct being measured | Retain distributions, multiple labels or population-specific judgments where useful | Erasing minority judgments through majority vote | Earlier aggregated labels may have lost information irreversibly unless the raw judgments were kept |

The taxonomy is adapted from a survey of learning from disagreement published in the Journal of Artificial Intelligence Research in 2021, which covers both NLP and vision datasets and identifies annotator or interface error, annotation-scheme problems, ambiguity, item difficulty and subjectivity as distinct sources. The adaptation to any one production task is a judgment, not a result.

Two rows deserve emphasis because they are the ones a delivery process most often collapses. The second row and the sixth row look identical on a dashboard, since both show two annotators who did not agree. They are opposites. One is a document defect that will keep producing disagreement until the document changes. The other is a property of the thing being measured, and forcing it to a single label destroys data.

## Not all label errors cost the same

If disagreement types differ, so does what they cost downstream, and there is controlled evidence for that.

Agnew and colleagues, publishing in IEEE Access in 2024, took subsets of COCO and ADE20K and synthetically introduced three annotation-error types: localization uncertainty, incorrect class labels, and missing annotations. They trained Mask R-CNN, YOLACT and Mask2Former variants across dataset sizes and degradation levels, 456 models in total. In the 100,000-image COCO object-detection condition, introducing incorrect class labels to 25, 50 and 75 percent of annotations reduced mAP, averaged across three models, by 0.024, 0.05 and 0.12 respectively. Across their setup, incorrect class labels had the largest adverse effect, followed by missing annotations and then localization uncertainty.

The limits are specific and they matter. The errors were deliberately induced into existing reference annotations, so this measured neither human annotator disagreement nor adjudication quality nor supplier accuracy, and the effect sizes belong to that experimental configuration.

The conclusion that does travel is narrow and useful: different label failure modes can have materially different downstream consequences. Which is why a single aggregate annotation-accuracy percentage is a weak procurement artifact. Ninety-seven percent tells a buyer very little until they know what the other three percent contains, and whether it is class confusions, missed instances, box geometry, identity switches or genuine ambiguity.

This is also the reason to be careful with agreement coefficients. Cohen's kappa measures chance-corrected agreement between two raters on nominal categories; it does not independently establish that the agreed category is correct. Work in Computational Linguistics in 2008 showed by simulation that a headline coefficient can hide whether disagreement is random or systematic, and that those two patterns can affect a learning problem very differently. Detection and tracking metrics such as mAP and HOTA are downstream system-evaluation measures rather than measures of annotator agreement, and using one in place of the other quietly changes the question. For the thresholds question specifically, our article on [data labeling QA thresholds](/blog/data-engineering/data-labeling-quality-assurance-thresholds/) covers what published sources actually support.

## Some disagreement does not go away when you add information

The instinct on an ambiguous item is to give annotators more context. That is sometimes right and it is not a general rule.

Pavlick and Kwiatkowski, in the Transactions of the ACL in 2019, sampled 100 premise and hypothesis pairs from each of five natural-language inference datasets and sought 50 independent ratings per pair, leaving 496 pairs with a mean of 39 labels each after filtering. In a separate context experiment, mean judgment variance rose from 0.34 at word level to 0.41 with sentences and 0.56 with paragraph context, each with a 95 percent confidence interval of roughly plus or minus 0.02. Adding context increased measured disagreement rather than resolving it.

That is a semantic inference task, and the numbers should not be transferred to bounding-box work. What transfers is the negative result: persistent disagreement cannot be universally diagnosed as insufficient instructions, insufficient annotators or insufficient context. Where interpretation is part of the construct, more information can expose differing human judgments instead of eliminating them.

The operational consequence is that calibration rounds are worth more as diagnosis than as a score. Have annotators independently label a controlled set, then compare decisions by failure mode rather than only by coefficient, and revise training or specification before scaling. The question is whether disagreements cluster around particular classes, edge cases or rules, and an aggregate number cannot answer it.

## Human review of a model pre-label is not independent review

Independent double annotation makes disagreement observable. A model pre-label followed by human approval does not, because the first label can influence the reviewer, and that difference is easy to lose in a workflow diagram where both look like "two passes."

A preregistered study at ACL Findings in 2025 by Schroeder, Roy and Kabbara tested this directly on subjective text coding, with 410 unique annotators and more than 7,000 annotations across three AI-assistance conditions, two models and two datasets. Annotators receiving model suggestions did not become faster, and they adopted the suggestions strongly enough to alter the resulting label distributions. On one dataset, where five independent crowd annotations aggregated by 3-of-5 majority formed the unassisted ground truth, average weighted F1 was 0.47 for one model and 0.44 for another. When the model-assisted crowd labels were used as ground truth instead, the reported weighted F1 was around 0.79.

That last comparison is the one to sit with. The apparent performance of the model rose substantially when the labels it was scored against had been produced with its own suggestions visible.

The study is complex subjective text coding, and the magnitude should not be quoted as an expected anchoring effect for bounding boxes. It does invalidate the general assumption that inserting a human confirmation step automatically restores independence. If your QA design depends on that assumption, it needs a different control, such as retaining a genuinely independent annotation stream on a sample.

## Finding defects is not the same as estimating how many there are

A risk-based review queue that prioritizes low-confidence items, rare classes, heavy occlusion and new annotators is an efficient way to find defects. Its raw defect percentage is not the dataset's error rate.

This is ordinary survey-sampling logic rather than anything specific to annotation: unequal inclusion probabilities require weighting before a sample supports a population-level estimate. A genuinely random audit estimates population defect prevalence with quantifiable sampling uncertainty. Stratified sampling is often more efficient for rare or operationally important slices, but a disproportionate sample needs suitable weights to say anything about the whole dataset.

Both instruments belong in a delivery process. Reporting the output of the first as though it were the output of the second is a reporting error, and it goes in both directions: a targeted queue overstates the defect rate to a buyer who assumes it was random, and a random audit understates how many defects a targeted pass would have found.

Published practice suggests this is not a rare confusion. A review in Computational Linguistics in 2024 analyzed 591 text-dataset publications and found common errors in the reported use of inter-annotator agreement and annotation-error rates, rating a substantial share of the studied work as subpar against the authors' framework. That corpus is NLP dataset papers rather than commercial delivery, so read it as evidence about measurement and reporting practice.

## When adjudication changes the specification

A specification change is not a record correction, and recording it as one loses the only information that makes the dataset auditable later.

The defensible sequence is to create a new ontology or guideline version, record the changed definition and the rationale for it, identify the records potentially produced under the old interpretation, decide whether they require re-review or re-annotation, recalibrate the affected annotators, and maintain provenance linking each accepted label to the specification version in force when it was made. This is an operating recommendation derived from the research rather than evidence that every project needs identical mechanics.

For a disputed item specifically, the evidence retained should be enough to reconstruct the decision without pretending the adjudicator established truth:

- The original annotations, unmodified.
- Annotator and reviewer identifiers, pseudonymous where required.
- The specification version in force at annotation time.
- A disagreement measure, where one is informative.
- The diagnosis, meaning which row of the table above applied.
- The adjudication decision and its rationale.
- The final accepted representation.
- Model or pre-label provenance, if the item passed through an assisted workflow.
- Any specification change, and the action taken on affected data.

A buyer reading that record can tell the difference between "one annotator made a mistake" and "our instructions did not cover this case until now." Those are the two claims most annotation deliveries fail to separate, and they have different implications for everything already delivered.

Every study cited here is third-party published work with its own task, dataset and stated limits. They support reasoning about mechanism. None of them supplies a threshold for your workload.

## YPAI managed annotation

YPAI runs annotation as an operating model rather than a labeling queue. Disagreement is diagnosed by cause before it is adjudicated, the specification is versioned when a rule changes rather than patched silently, and adjudication decisions are recorded with the evidence needed to reconstruct them.

Data and evaluation leads with a defined annotation workload can [request a consultation](/contact-us/), or read how we structure [managed annotation](/data-solutions/annotation/).

---

## Related Resources

- [Data labeling QA thresholds](/blog/data-engineering/data-labeling-quality-assurance-thresholds/) - What published sources support on agreement metrics and audit rates
- [Audio annotation pipeline for speech data labeling](/blog/data-engineering/audio-annotation-pipeline-speech-data-labeling/) - Stage-by-stage QA gates in a production annotation pipeline
- [Data annotation pricing enterprise guide](/blog/data-engineering/data-annotation-pricing-enterprise-guide/) - What adjudication and multi-pass review actually cost
- [EU AI Act Article 10 engineering requirements](/blog/compliance/eu-ai-act-article-10-engineering-requirements/) - Data-governance obligations and what they mean for engineering teams
- [Managed annotation](/data-solutions/annotation/) - Ontology, calibration and adjudication with YPAI