The shift from high-volume labeling to expert human feedback

Why annotation evolved beyond bounding boxes

Here's the thing, when most people hear "AI annotation," they still picture armies of contractors clicking bounding boxes around street signs. That was 2016. Frontier labs today need something fundamentally different: expert judgment that shapes how models reason, refuse requests, and handle edge cases.

The shift happened because model capability outpaced simple supervised learning. Once language models could generate fluent text, the bottleneck became teaching them which answer is better and why, questions that require domain expertise, ethical reasoning, and nuanced preference judgments. From what I've seen, companies like Scale AI went from a high-volume image-labeling provider to essential infrastructure for OpenAI and Anthropic precisely because they could recruit credentialed professionals to write those judgments.

This isn't a small shift in task design. High-quality RLHF annotations now cost roughly 100x more per item than traditional labeling, with domain specialists commanding $50-200/hour. The economics flipped: you're no longer optimizing for volume, you're competing for the medical doctor who can explain why one clinical reasoning chain is safer than another.

Check your understanding

A PM at a startup building medical AI wants to improve model accuracy on diagnosis tasks. They're debating between collecting 50,000 more image labels from generalist annotators at $0.50 each, or 600 preference rankings from licensed physicians at $100 each. Both cost about $25k–$60k. Which choice best reflects the modern annotation paradigm?

RLHF is not just thumbs up and thumbs down

Let's get real, RLHF (Reinforcement Learning from Human Feedback) sounds simple in slide decks, but it's a multi-stage technical pipeline that transforms human preferences into model behavior. ChatGPT, Claude, and Gemini all use it as a core training method, and the implementation details matter more than most PMs realize.

The workflow has three distinct phases. First, human annotators rank model outputs through pairwise comparisons, "is response A or B better for this prompt?" These rankings become training data. Second, you train a separate reward model that learns to quantify those preferences into numerical signals. Third, you use reinforcement learning (typically an algorithm called PPO) to optimize the language model policy so it generates outputs that score higher on the reward model.

Here's where it gets messy. Human annotators disagree, sometimes fundamentally, about which output is "better." Reward models can be gamed through reward hacking, where the model finds shortcuts that maximize the score without actually improving helpfulness. Nathan Lambert's research shows that iterative RLHF training can cost millions, with meaningful portions spent on data that doesn't make the final cut. From a PM lens, you're managing disagreement resolution workflows, quality calibration, and the reality that expert feedback is both your most valuable signal and your biggest bottleneck.

The reward model is the translation layer

Check your understanding

Match each RLHF stage to its primary function in the pipeline:

Red teaming and Constitutional AI: teaching models what not to do

Teaching a model to be helpful is hard. Teaching it to refuse harmful requests while staying useful is exponentially harder, and that's where red teaming and Constitutional AI come in.

Red teaming for AI isn't traditional penetration testing. It targets unique attack surfaces of ML systems: prompt injection, jailbreaks, training data poisoning, emergent behaviors that weren't in any training example. Scale AI was selected by the White House to conduct public assessments of leading models precisely because they'd built libraries of adversarial prompt sets and trained thousands of red teamers to systematically probe for vulnerabilities. From what I've observed, effective red teaming combines security testing (can you extract training data?) with safety testing (will it generate instructions for harm?).

Constitutional AI, pioneered by Amanda Askell and the Anthropic team, takes a different approach. Instead of opaque reward functions, you embed explicit ethical principles and reasoning frameworks directly into training. Claude was trained with a 23,000-word constitution derived from sources like the Universal Declaration of Human Rights, establishing a priority hierarchy: safety > ethics > compliance > helpfulness. The key insight: rather than hardcoding rules, you teach the model to critique and revise its own outputs according to defined values, so it constructs appropriate responses by understanding underlying principles.

For you as a PM, this matters because both approaches require structured expert workflows at scale. Red teaming needs adversarial creativity and domain knowledge. Constitutional AI needs philosophers and ethicists who can articulate why a principle applies. You're no longer just shipping labeling tasks, you're designing feedback loops that encode human values into billion-parameter systems.

Check your understanding

Meta's annotation insurance play

Evaluation is now its own discipline

Here's something that surprised me when I dug into production AI systems: evaluation is often harder than training. Hamel Husain and researchers like Dr. Craig Martell have been shouting this from the rooftops, you can't evaluate AI quality with generic metrics like "hallucination_score" or "helpfulness_score."

Effective evaluation requires domain-specific rubrics, application-specific criteria, and often expert human judgment. Foundation model benchmarks (MMLU, HumanEval) tell you almost nothing about whether your customer support AI handles refund edge cases correctly. You need structured assessment workflows: rubric-based scoring where experts rate outputs on 5-10 dimensions, side-by-side comparisons to detect regression, human-in-the-loop review for high-stakes decisions.

The infrastructure evolution mirrors the shift in annotation work. Early platforms were simple labeling interfaces. Modern systems support query sampling, feedback translation, multi-layer review workflows, quality calibration, and disagreement resolution at scale. You're not just collecting data, you're orchestrating expert judgment pipelines that determine whether billion-dollar training runs advance capabilities or optimize the wrong objectives. (That last part keeps me up at night.)

Key takeaways

• Annotation evolved from high-volume commodity labeling to expert human feedback that shapes model reasoning, refusal behavior, and ethical judgment.
• RLHF is a three-stage technical pipeline, preference collection, reward model training, and RL optimization, not just thumbs up/thumbs down.
• Red teaming targets ML-specific attack surfaces like prompt injection and jailbreaks, while Constitutional AI embeds reasoning frameworks so models critique their own outputs.
• High-quality expert annotations now cost ~100x more per item than traditional labels, with domain specialists commanding $50-200/hour.
• Evaluation requires domain-specific rubrics and expert review, foundation model benchmarks are insufficient for production use cases.

Your product check-in

Apply “The shift from high-volume labeling to expert human feedback” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?