Output metrics: precision/recall, IAA, rejection rate, time-to-delivery

What output metrics actually measure

Here's the thing, output metrics tell you what you delivered, not what you controlled. They're the lagging indicators of your annotation pipeline's quality, showing up after labelers finish, after QA runs, after customers accept or reject the batch. From what I've seen in platforms like Scale and Appen, directors who treat output metrics as post-mortems rather than real-time levers miss half the story.

The four metrics that matter most are inter-annotator agreement (IAA), precision/recall against gold standards, rejection or rework rates, and time-to-delivery. IAA tells you if your task definition is coherent. Precision and recall measure false positives and false negatives before training even begins. Rejection rate captures the invisible cost of wasted work. Time-to-delivery reveals whether you can hit the training windows that model teams depend on.

These metrics sit in the middle layer of your metric tree: inputs drive them, and they predict model impact. A modest label-error rates in ground-truth data can meaningfully degrade final model accuracy (Waymo's experience with Scale Nucleus shows this in safety-critical LiDAR annotation). You're not just measuring quality for its own sake, you're predicting whether the data will make the model better.

Check your understanding

Your model training pipeline is delayed because the latest annotation batch arrived on time but was rejected by the ML team for low quality. Which output metric would have flagged this issue earliest?

Inter-annotator agreement: when disagreement is the signal

IAA measures consistency across labelers on the same data. The two standards you'll see most are Cohen's Kappa for simple categorical tasks and Krippendorff's alpha for complex, structured annotation like bounding boxes or multi-label NLP. Ron Artstein's 2008 survey became the definitive guide here, and Klaus Krippendorff's distance-based metric handles the nuance that Scale's enterprise customers need.

The thresholds matter. IAA above 0.8 is good. Below 0.4 requires intervention. But here's the nuance directors miss: low IAA doesn't always mean bad annotators. It often reveals ambiguous guidelines, unclear task definitions, or inherent complexity in what you're asking people to label. I've seen teams add consensus workflows when the real fix was rewriting the instructions.

High IAA also isn't always good news. If your task is so simple that agreement is trivial, you're probably not capturing the edge cases or subjective judgments that frontier AI models need. (OpenAI and Anthropic's RLHF pipelines balance agreement with the need for diverse human preferences, the disagreement can be valuable signal about what's actually ambiguous in human values.)

Low IAA is a task design smell

Check your understanding

For most annotation tasks, inter-annotator agreement above ___ is considered good, while agreement below 0.4 signals the need for immediate intervention.

Precision, recall, and the gold standard reality check

Precision and recall aren't just model metrics, they're fundamental annotation quality metrics measured against gold standards before any training happens. Precision tells you how many of the labels marked "positive" are actually correct (false positive rate). Recall tells you how many true positives you captured (false negative rate). Both matter, and the tradeoff depends on your use case.

Appen inserts gold set items at 5-10% of total volume, randomly mixed into annotator queues. If an annotator's precision or recall drops below your threshold (often 80% for qualified annotators), you catch it in real-time, not after you've shipped a broken batch. This is how platforms like Scale maintain the quality their customers pay $48/hour for, they measure against known-answer items continuously.

The challenge is creating high-quality gold sets in the first place. For ambiguous tasks like "Is this response helpful?" or safety classification, you need expert review and consensus to establish ground truth. (Surge AI's model of recruiting domain experts, lawyers, doctors, coders, is built on the idea that gold sets for frontier AI can't be crowdsourced from generalists.)

70% of model errors start in the data

Check your understanding

Rejection rate and time-to-delivery: the invisible costs

Rejection rate is the percentage of delivered work that customers or internal reviewers send back for rework. It's the most direct measure of wasted effort in your pipeline. High rejection rates create invisible costs, annotator time, QA cycles, delayed model training, and frustrated customers who planned their release around your SLA.

Here's what I've observed: teams often optimize for completion rate ("we labeled 100,000 examples this week!") when acceptance rate is the metric that predicts revenue and renewal. If 30% of your delivered batches get rejected, you didn't ship 70,000 useful labels, you shipped 70,000 and wasted resources on 30,000 that now need rework.

Time-to-delivery matters just as much as quality. SLA compliance, typically under 24 hours for top annotation teams, determines whether your data arrives in time for training windows and model releases. Late high-quality data misses the pipeline. From what I've seen at platforms like Scale, both dimensions must be balanced: you can't sacrifice speed for quality or vice versa. The best teams instrument both and surface the tradeoff explicitly in their metric tree.

Track acceptance, not just completion

Key takeaways

• Output metrics are lagging indicators, they show what you delivered (IAA, precision/recall, rejection rate, time-to-delivery) after the work is done.
• Inter-annotator agreement above 0.8 is good; below 0.4 usually signals ambiguous guidelines or task definition problems, not just bad annotators.
• Precision and recall measure annotation quality against gold standards before training, a 10% error rate in labels can reduce model accuracy by 15%.
• Rejection and rework rates reveal the invisible cost of wasted effort; acceptance rate predicts customer value better than raw completion volume.
• Time-to-delivery and quality must both be balanced, late high-quality data misses training windows, but fast low-quality data wastes downstream model development.

Your product check-in

Apply “Output metrics: precision/recall, IAA, rejection rate, time-to-delivery” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?