Red teaming, adversarial testing, and model evaluation tasks

When annotation becomes adversarial, why frontier labs pay experts to break their models

Here's the thing, the annotation work that shapes frontier models isn't about labeling images anymore. It's about systematic attempts to make models fail. Red teaming for AI means simulating real-world attacks: prompt injection, jailbreaks, bias amplification, generating harmful content. Scale AI was selected by the White House to conduct public assessments of leading models, deploying thousands of trained red teamers with extensive libraries of adversarial prompts to scan for vulnerabilities.

This isn't traditional penetration testing. AI red teaming targets unique attack surfaces of ML systems, training data poisoning, inference pipeline exploits, emergent behaviors that weren't in the training set. You're testing both security (can someone make this do something malicious?) and safety (will this cause unintended harm in the wild?). From what I've seen, the distinction matters: a model might pass security audits but still produce biased hiring recommendations or unsafe medical advice.

The economics tell the story. Expert red teamers command $50-200/hour, and a single high-quality adversarial test suite can cost tens of thousands to develop. That's because you need people who understand both the domain (medical reasoning, legal advice, child safety) and the technical attack vectors. This work doesn't scale through volume, it scales through expertise.

Check your understanding

A VP asks you to budget for red teaming their new customer service chatbot. What combination of work should you plan for?

Model evaluation as a product discipline, rubrics, side-by-sides, and the analyze-measure-improve loop

Let's get real: you can't ship a model with generic metrics like 'helpfulness_score' and call it done. Effective evaluation requires domain-specific rubrics, application-specific criteria, and often expert human judgment. Hamel Husain has codified this as the analyze-measure-improve cycle for production AI, you define what good looks like for your use case, instrument measurement workflows, and iterate.

In practice, this means structured assessment workflows: rubric-based scoring (does this medical summary include contraindications?), side-by-side comparisons (which response better handles the customer's frustration?), domain benchmarks (can this code model pass our internal security standards?). ChatGPT, Claude, and Gemini all rely on multi-layer human review before deployment, with disagreement resolution protocols when raters diverge.

Dr. Craig Martell has argued for independent third-party evaluation because gaming leaderboards makes it hard to tell if systems actually improve. From what I've observed, the annotation platform becomes the evaluation infrastructure, query sampling, feedback translation, quality calibration, all at scale. You're not just collecting data; you're building the measurement apparatus that determines whether a billion-dollar training run succeeded.

The bottleneck shifted from quantity to quality

Check your understanding

Constitutional AI and the shift from opaque rewards to reasoning frameworks

Here's where things get interesting. Traditional RLHF trains a reward model that scores outputs numerically, but that number is opaque. You don't know why response A scored 4.2 and response B scored 3.8, which makes debugging and alignment incredibly hard. Amanda Askell at Anthropic pioneered Constitutional AI as an alternative: embed explicit ethical principles and reasoning frameworks directly into training.

Anthropic's Claude is trained with a 23,000-word constitution derived from sources like the Universal Declaration of Human Rights, with a 4-tier priority hierarchy: safety > ethics > compliance > helpfulness. The model critiques its own outputs according to these defined values, constructing responses by reasoning through underlying principles rather than following rigid rules. (This is subtle but crucial, it's not hardcoded behavior, it's learned reasoning about values.)

From what I've seen, this matters for product work because it makes model behavior more interpretable and steerable. When something goes wrong, you can trace it back to which constitutional principle was violated or misweighted. You can update the constitution without retraining from scratch. It's the difference between tuning an opaque function and editing a policy document your model can read.

Check your understanding

Your company is building a healthcare AI assistant. A PM proposes using Constitutional AI instead of traditional RLHF. What are two concrete advantages this approach would give your team when iterating on model behavior?

RLHF is not just thumbs-up/thumbs-down

The task taxonomy, from bounding boxes to chain-of-thought critique

The progression matters. Simple annotation types, image classification, bounding boxes, transcription, are high-volume, low-expertise work that can be done at scale for pennies per task. Complex structured tasks, preference ranking, chain-of-thought reasoning, critique generation, multi-object annotation with relationships, require domain specialists and cost 100x more per item.

Scale AI's evolution tells the story: founded in 2016 as an image annotation platform, now a roughly $870M revenue provider of expert feedback for OpenAI, Anthropic, and frontier labs. They recruit credentialed professionals to design evaluation rubrics and generate preference data, not just execute labeling instructions. Medical reasoning models need feedback from licensed physicians; legal AI needs practicing attorneys. The annotation platform becomes a talent marketplace for specialized judgment.

From what I've observed, the product challenge is building infrastructure that supports this complexity: query sampling strategies that surface edge cases, multi-layer review workflows, disagreement resolution at scale, quality calibration across expert pools. Hummingbird Technologies partnered with CloudFactory for agricultural AI, moving from traditional annotation to expert-level analysis where annotators identify subtle crop health indicators invisible to non-experts. That requires different tooling, different economics, different quality controls.

Key takeaways

• Red teaming AI systems targets unique ML attack surfaces, prompt injection, jailbreaks, emergent behaviors, and combines security testing with safety testing to prevent both exploitation and unintended harm.
• Effective model evaluation requires domain-specific rubrics and expert judgment; foundation model benchmarks are insufficient for production use cases.
• Constitutional AI embeds reasoning frameworks that explain ethical principles, making model behavior more interpretable and steerable than opaque RLHF reward scores.
• The annotation task taxonomy has progressed from high-volume commodity labeling to complex structured tasks requiring credentialed domain experts at 100x the cost.
• The bottleneck for frontier models shifted from data quantity to quality, expert human judgment is now critical infrastructure, not a temporary scaffold.

Your product check-in

Apply “Red teaming, adversarial testing, and model evaluation tasks” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?