Task taxonomy: classification, segmentation, transcription, rating, ranking, chain-of-thought, critique

From button-clicking to expert judgment

Here's the thing most people miss about annotation work. When Scale AI launched in 2016, the core business was bounding boxes and image classification, high-volume, commodity tasks that paid pennies per item. By 2025, they're projected to pull roughly $2B in revenue by recruiting credentialed professionals (MDs, JDs, PhDs) to generate expert feedback for frontier model training. The shift isn't subtle: RLHF annotations now cost ~100x more per item than traditional labeling.

The taxonomy of annotation tasks maps directly to this economic evolution. On one end, you have simple classification and transcription, the kind of work you can scale to thousands of annotators with minimal training. On the other, you have chain-of-thought reasoning, critique generation, and preference ranking that require domain expertise and nuanced judgment. As models get more capable, they automate the commodity tasks but become more dependent on expert human input.

From what I've seen, the director-level PM mistake is treating all annotation as the same workflow problem. The infrastructure, economics, and quality playbook for transcription are fundamentally different from what you need for constitutional AI feedback. Let's walk through the taxonomy so you can spot which task type you're dealing with, and what that implies for platform design.

Check your understanding

Match each annotation task type to its defining characteristic:

Rating and ranking: the RLHF backbone

Let's talk about the tasks that actually train ChatGPT, Claude, and Gemini. RLHF (Reinforcement Learning from Human Feedback) starts with a deceptively simple annotation workflow: show an annotator two model responses to the same prompt, ask them to pick which one is better. These pairwise preference rankings become the training signal for a reward model, a numerical function that quantifies human preferences.

The nuance shows up when you realize this isn't just thumbs-up-thumbs-down button clicking. Annotators are making multi-dimensional judgments: Is this response more helpful? More honest? Less likely to cause harm? Nathan Lambert's RLHF research shows that human disagreement rates on preference tasks can hit 30-40% for edge cases, and those disagreements carry real training consequences, the reward model learns whatever pattern you feed it, including your annotators' biases and blind spots.

Rating tasks are the simpler cousin: score a single output on a rubric (1-5 stars, pass/fail, or multi-axis evaluation). You see this in model evaluation workflows where domain experts assess outputs against specific criteria before deployment. The difference matters for platform design, ranking requires careful sampling to show comparable outputs, while rating needs rock-solid rubric design and calibration to prevent score drift across annotators.

Why pairwise comparison beats absolute scoring

Check your understanding

You're designing an RLHF annotation workflow for a customer service chatbot. Annotators report that 35% of the time, both model responses seem equally good or equally bad, making it hard to choose. What's the best product decision?

Chain-of-thought and critique: teaching models to reason

This is where annotation crosses into a different domain entirely. Chain-of-thought (CoT) tasks ask annotators to write out the intermediate reasoning steps that lead to an answer, not just 'the capital of France is Paris,' but 'France is a European country, its capital is the seat of government, historically that has been Paris since...' The goal is to teach models to show their work, making outputs more interpretable and reducing hallucination.

CoT annotation is expensive because it requires legible expert reasoning. You can't offshore this to low-cost labor pools, you need people who actually have the domain knowledge and can articulate their thought process clearly. For medical reasoning models, that means licensed physicians generating step-by-step diagnostic logic. For legal analysis, it's attorneys walking through case law application. Amanda Askell's work at Anthropic on crafting Claude's reasoning style shows how much craft goes into this: it's not just correct logic, it's pedagogically useful logic that helps the model generalize.

Critique tasks flip the script: instead of generating reasoning, annotators evaluate model-generated reasoning. Constitutional AI relies heavily on this, models produce an initial response, then critique it against a defined set of principles (safety, ethics, compliance, helpfulness), and generate a revised version. Anthropic's 23,000-word constitution for Claude creates a structured framework for these critiques, establishing a 4-tier priority hierarchy so the model knows safety concerns override helpfulness optimization.

Scale AI's red teaming for the White House

Check your understanding

Key takeaways

• Annotation tasks span a complexity spectrum from simple classification to expert critique, each requires fundamentally different platform infrastructure and economics.
• RLHF workflows rely on pairwise preference ranking to train reward models, and high-quality implementations capture ties and annotator disagreement as legitimate signal.
• Chain-of-thought and critique tasks require credentialed domain experts who can articulate legible reasoning, costing ~100x more per item than commodity labeling.
• As models automate simple tasks, dependence on expert human judgment intensifies rather than diminishes, the bottleneck shifted from data volume to annotation quality.
• Task taxonomy determines your platform design: transcription needs throughput optimization, constitutional AI critique needs rubric calibration and multi-stage review workflows.

Your product check-in

Apply “Task taxonomy: classification, segmentation, transcription, rating, ranking, chain-of-thought, critique” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?