Requester/customer surface: dataset creation, project setup, quality SLAs, and data delivery

What the requester actually needs to build AI

Here's the thing, when an ML team comes to your annotation platform, they're not just buying labels. They're buying a reproducible pipeline that turns messy data into training sets their models can learn from. That means project setup, ontology definition, quality guarantees, and delivery formats that plug directly into their training infrastructure.

From what I've seen, the requester surface is where most annotation platforms either win or lose enterprise deals. Customers need transparent quality metrics, accuracy rates, inter-annotator agreement scores, IoU thresholds for bounding boxes, and they need delivery timelines they can build a sprint plan around. If they can't see what they're getting or when, they'll build in-house or move to a competitor.

The tricky part is that speed, cost, and quality form an iron triangle. Requesters want all three, but your platform architecture forces trade-offs at every layer, workforce routing, review cycles, consensus workflows. Your job as PM is to make those trade-offs explicit and give customers levers to tune them.

Check your understanding

An ML team at an autonomous vehicle company wants to launch a new perception model in eight weeks. They need 50,000 labeled images with complex occlusion cases. The team lead asks you to prioritize speed over cost. What should you recommend?

Project setup: ontologies, SLAs, and the delivery contract

Every annotation project starts with ontology definition, the taxonomy of labels, attributes, and hierarchies that map onto the customer's model architecture. This isn't just a config file. It's a governance artifact that needs versioning, change tracking, and audit trails. When a customer updates a label definition mid-project, downstream systems have to handle backward-incompatible changes, and you need to re-train annotators on the new guidelines.

Quality SLAs are where the rubber meets the road. Industry minimums are Cohen's Kappa >0.85 for classification tasks and IoU >0.80 for bounding boxes, but customers building safety-critical systems (autonomous vehicles, medical imaging) often demand higher thresholds. Alexandr Wang built Scale AI's reputation by guaranteeing those numbers and logging every label decision with timestamps and annotator IDs for auditability.

Delivery formats matter more than most PMs realize. Your platform needs to export in formats that plug directly into ML pipelines, COCO JSON, Pascal VOC, TFRecord, or custom schemas. If the ML engineer has to write transformation scripts, you've added friction that costs you the renewal.

Labelbox's integration strategy

Labelbox won enterprise deals by building API connections directly into ML training pipelines. Their platform exports annotations in standard formats and provides webhooks for real-time delivery as batches complete. This reduces the time from annotation to model training from days to hours, a key differentiator when customers are iterating on model architectures weekly.

Check your understanding

A customer requests a mid-project ontology change: they want to split the existing 'pedestrian' label into 'pedestrian_adult' and 'pedestrian_child' for better model granularity. You have 20,000 images already labeled and 30,000 remaining. What factors should you consider before approving this change, and how would you approach the transition?

Quality transparency: showing your work

Let's get real, customers don't trust black-box quality claims. If you tell them "95% accuracy" without showing how you measured it, they'll assume you're gaming the metric. The requester surface needs real-time dashboards that surface inter-annotator agreement, gold standard accuracy, review cycle pass rates, and edge case coverage.

From what I've observed, the platforms that win enterprise trust use multi-layer quality control: gold standard datasets (200-500 examples with documented rationale), honeypot validation mixed into live tasks, consensus voting on 10% overlap, and review workflows where expert annotators adjudicate disagreements. SuperAnnotate cut annotation cycle time by 60% by making these metrics visible and actionable for customers.

Here's the nuance Milagros Miceli's research highlights: quality metrics can also become surveillance tools that harm annotators. Your platform needs to balance customer transparency with workforce fairness, showing aggregate quality without creating punitive individual tracking that pressures annotators into rushed, lower-quality work to hit throughput targets.

Check your understanding

Key takeaways

Requesters buy reproducible pipelines, not just labels, project setup includes ontology versioning, quality SLAs, and delivery formats that plug into ML training infrastructure.
Quality SLAs must be transparent and measurable: Cohen's Kappa >0.85 for classification, IoU >0.80 for bounding boxes, with real-time dashboards showing inter-annotator agreement and gold standard accuracy.
Ontology changes are governance problems requiring versioning, annotator re-training, backward compatibility, and SLA adjustments, not just config updates.
Speed, cost, and quality form an iron triangle, your job as PM is to make trade-offs explicit and give customers levers to tune them.
Multi-layer quality control (gold sets, honeypots, consensus, review workflows) builds enterprise trust while balancing customer transparency with workforce fairness.

Your product check-in

Apply “Requester/customer surface: dataset creation, project setup, quality SLAs, and data delivery” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?