Anatomy of a Good Rubric: Criteria, Scales, and Examples

What Makes a Rubric Actually Work

Here's the thing, when I first started building AI evals, I thought a rubric was just a fancy word for "rate it 1 to 5." What I learned the hard way: a good rubric is a decision tree that multiple people can walk down and land in the same place. If two evaluators look at the same model output and give wildly different scores, your rubric isn't a rubric, it's a Rorschach test.

A working rubric has four essential components: evaluation dimensions (what you're measuring, accuracy, helpfulness, safety), score levels (the anchors, like pass/fail or 1-5), behavioral descriptions (concrete examples for each score, not vague labels), and weighting systems (which dimensions matter most). Miss any of these and you're back to vibes.

The real test? Inter-rater agreement. Aman Khan, who developed the popular evals course with Andrew Ng, puts it bluntly: if your Krippendorff's alpha isn't near 0.8, you don't have a rubric yet, you have a conversation starter. From what I've seen, most teams skip this validation step and wonder why their eval pipeline produces noise instead of signal.

Check your understanding

Match each rubric component to what it defines:

Binary vs. Likert: Why Simpler Often Wins

There's a persistent belief that more granularity equals better measurement. I've watched teams design elaborate 7-point scales, convinced they're capturing nuance. Then they look at the data: evaluators cluster around 4 or 5, adjacent points blur together, and inter-rater agreement tanks.

Hamel Husain has been beating this drum for years: binary pass/fail evaluations force clearer thinking. When you can only say "yes" or "no," you have to define the threshold precisely. You need smaller sample sizes to detect real differences. And crucially, two evaluators are far more likely to agree on whether something passed than on whether it's a 3 versus a 4.

That said, Likert scales (typically 1-5) do have a place, when you genuinely need to distinguish levels of performance and you've defined behavioral anchors for every point. The key word is behavioral. "Response addresses core question but omits two relevant examples" is an anchor. "Adequate" is not. If you can't write a concrete description for each score level, collapse it to binary.

The Central Tendency Trap

Likert scales suffer from central tendency bias, evaluators gravitate toward middle scores (3 on a 5-point scale) to avoid seeming too harsh or too lenient. This compresses your data and hides real differences. Binary forces a decision.

Check your understanding

You're building an eval for a customer support chatbot. Initial testing with a 5-point helpfulness scale shows evaluators mostly scoring 3s and 4s, with low agreement on adjacent scores. What's your best next move?

Task-Specific Dimensions: One Size Fits Nobody

Here's where most generic rubrics fall apart. A customer support chatbot needs to measure empathy, escalation handling, and policy accuracy. A code generation model? You're looking at correctness, efficiency, readability, and security. Trying to evaluate both with the same "rate quality 1-5" rubric is like using a ruler to measure temperature.

Look at how Meta evaluated LLaMA-2: they built separate dimensions for helpfulness, truthfulness, and harmlessness. A model can be incredibly helpful while being dangerously untruthful, you need both axes. Emily Reese and the Scale AI research team took this further with their Rubrics as Rewards framework, showing that explicit criteria (what facts to include, what reasoning steps matter, what pitfalls to avoid) plus semantic weighting (Essential vs Important) improved model training outcomes by 28% over simple Likert baselines.

From what I've observed, the best product teams start by asking: what downstream outcome am I trying to predict? For a health app like Fitbit's Insights Explorer, that meant five dimensions, Safety, Helpfulness, Accuracy, Relevance, and Personalization, validated by both generalists and clinical specialists. Your rubric should map directly to what success looks like for your specific use case, not some abstract notion of "good."

Check your understanding

A good rubric typically achieves a Krippendorff's alpha near ___ to demonstrate high inter-rater agreement.

Rubric Calibration Is Not Optional

Before you scale to LLM-as-judge or large eval runs, validate your rubric with human-labeled ground truth. Check inter-rater agreement, confirm that scores discriminate between good and bad outputs, and verify that your dimensions actually predict downstream outcomes. Skipping calibration means you're optimizing for noise.

Key takeaways

A working rubric has four components: evaluation dimensions, score levels, behavioral descriptions, and weighting, miss any and you're measuring vibes.
Binary pass/fail often beats multi-point scales because it forces clear thresholds, produces higher agreement, and needs smaller samples.
Behavioral anchors are concrete examples for each score, "addresses core question but omits two examples" not "adequate."
Task-specific dimensions matter: customer support rubrics measure empathy and escalation; code rubrics measure correctness and security.
Validate your rubric with inter-rater agreement (Krippendorff's alpha near 0.8) before scaling to automated evals.

Your product check-in

Apply “Anatomy of a Good Rubric: Criteria, Scales, and Examples” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?