The workflow journey: before (scoping, recruiting, task design), during (labeling, QA, adjudication), after (delivery, model impact, feedback)

The three-act structure of annotation workflow

Here's the thing about annotation platforms, they're not just tools, they're orchestration engines. Every annotation project moves through three distinct phases: before you launch (scoping requirements, recruiting the right people, and designing task guidelines that don't fall apart on day two), during execution (labeling at scale, multi-layer QA, and adjudicating the inevitable disagreements), and after delivery (packaging the dataset, measuring downstream model impact, and closing the feedback loop for the next round).

Most PM interviews trip people up on the handoffs between these phases. A requester from OpenAI wants GPT-level RLHF feedback, nuanced, context-aware, justifying why one response beats another. If you recruit general crowd workers at $15/hr without domain expertise, your during-phase QA becomes a nightmare. If you don't build feedback loops in the after phase, you never learn which edge cases broke the model, and your next project repeats the same mistakes.

From what I've seen, the best platforms don't just provide tooling for each phase, they design handoffs that preserve context. When Alexandr Wang talks about Scale AI as infrastructure, he means the workflow memory: task definitions that flow into real-time QA dashboards, consensus data that surfaces in post-delivery analytics, model performance signals that inform the next recruiting brief.

Check your understanding

Match each workflow phase to the key activity that defines it.

Before: the hidden cost of bad scoping and recruiting

Let's get real, 80% of ML engineering effort goes to data prep, and most of that pain starts in the before phase. A requester walks in with "we need 50,000 annotations for autonomous vehicle perception." Sounds straightforward. Then you discover they need LiDAR 3D point cloud alignment, understanding of sensor fusion, and annotators who know the difference between SAE Level 2 and Level 3 autonomy. That's not a $15/hr crowd task, that's a $40-100/hr expert marketplace play.

The recruiting decision cascades. If you source from a general crowd (like early APPEN projects), you'll hit high turnover and inconsistent quality that your QA layer can't fix. If you over-index on credentialed experts without task design that keeps them engaged, you burn budget and still get mediocre output. Braintrust's AI-powered vetting solved part of this, onboarding 4,000+ vetted contributors monthly with <72 hour ramp-up, by treating recruiting as a product problem, not a procurement afterthought.

Task design is where theory meets reality. Guidelines that work on paper fall apart when annotators encounter edge cases you didn't anticipate. Sama's model, operating annotation centers in Kenya, Uganda, and India with long-tenured, trained staff, demonstrates that investing in workforce stability during the before phase pays off in the during phase. Annotators who stay for years internalize nuance that no 50-page guideline can capture.

The anonymous crowd trap

Check your understanding

You're scoping an RLHF annotation project for a conversational AI model that needs to understand legal risk in contract clauses. The requester wants 20,000 preference judgments (A vs. B response quality) within 4 weeks. Which recruiting approach best balances quality and timeline?

During: QA is not an afterthought, it's the workflow

From what I've observed, the difference between platforms that scale and platforms that collapse under their own weight is whether QA is built into the during phase or bolted on after. Production annotation integrates quality controls throughout, gold-set monitoring (pre-labeled test samples sprinkled into live tasks), consensus-based review (multiple annotators per item, flagging disagreement), and real-time anomaly detection (catch an annotator who's clicking randomly before they ruin 1,000 labels).

SuperAnnotate reported customers saw 60% reduction in annotation cycle time and 35% increase in reviewer throughput by unifying annotators, reviewers, and project managers in a single system with real-time consensus tracking. That's the product insight: when QA reviewers can see inter-annotator agreement live and jump in to adjudicate edge cases immediately, you prevent drift. When they only see aggregated data at the end of the week, the damage is done.

The adjudication workflow is where most platforms get stuck. You've got Cohen's Kappa or Fleiss' Kappa scores (>0.80 considered strong agreement, per Ron Artstein and Massimo Poesio's foundational work), but what happens when two expert annotators genuinely disagree on a subjective call? You need a senior reviewer, often someone with more domain depth, to break the tie. That expert adjudication layer typically consumes 15-25% of your total annotation budget. Go above 25% and you've got a process problem, not a quality problem.

Quality comes from design, not headcount

Check your understanding

You're designing the during-phase QA workflow for a medical imaging annotation project (identifying tumors in radiology scans). Three radiologists will annotate each scan independently, and you expect moderate disagreement on edge cases (small lesions, ambiguous shadows). Describe how you would structure the consensus review and adjudication process to maintain quality without blowing your QA budget.

After: closing the loop from delivery to model impact

The after phase is where most annotation platforms leave value on the table. You ship the dataset, invoice the requester, and move on. But the requester is about to discover which annotations actually mattered for model performance, which edge cases broke the model, which labeling errors the model learned anyway, which high-confidence samples were wasted effort.

Here's the product opportunity: build feedback loops that capture model impact and route it back into the next project's before phase. Scale AI's hybrid human-plus-model pipeline does this implicitly, when their automation pre-labels the easy 80% and routes the hard 20% to humans, they're using model confidence signals from prior projects to inform task design. Meta's ~$14.3B investment for a 49% Scale stake in June 2025 wasn't just about volume, it was about annotation infrastructure as a strategic moat, the platform that learns fastest from deployment feedback compounds its advantage.

Versioned dataset lineage tracking is the unglamorous but essential piece. When a requester reports that their model hallucinates on a specific prompt type, can you trace back to the annotation cohort, the guidelines version, and the reviewers who touched those samples? If not, you're flying blind. If yes, you close the loop: update guidelines, retrain that annotator cohort, and flag similar cases in future projects. That's when annotation becomes infrastructure, not just a service.

Interview signal: talk about feedback loops

Key takeaways

• Annotation workflow splits into three phases, before (scoping, recruiting, task design), during (labeling, QA, adjudication), after (delivery, model impact, feedback), and the handoffs between them determine platform quality.
• The before phase recruiting decision cascades: domain expert tasks ($40-100/hr) require vetted marketplaces, not anonymous crowds; bad recruiting creates QA nightmares you can't fix later.
• During-phase QA must be real-time and integrated, gold-set monitoring, consensus review, and expert adjudication built into the workflow, with 15-25% of budget; more than 25% signals a process problem.
• Inter-annotator agreement (Cohen's Kappa >0.80) is the operational metric for quality, but it's a lagging indicator, leading indicators are calibration quality and guideline clarity before work starts.
• The after phase unlocks compounding advantage: platforms that close the loop from model performance back to recruiting and task design learn faster than competitors and become infrastructure, not just vendors.

Your product check-in

Apply “The workflow journey: before (scoping, recruiting, task design), during (labeling, QA, adjudication), after (delivery, model impact, feedback)” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?