Binary vs. Likert vs. Custom Scales: When to Use Each

The scale you choose shapes the thinking

Here's the thing, before you write any rubric, you need to decide how many levels of performance you're going to capture. Binary (pass/fail)? A 5-point Likert scale? Something custom? From what I've seen, most teams default to a 1-5 rating because it feels more nuanced. But that assumption can actually introduce more noise than signal.

The scale you pick determines how clearly your evaluators think, how consistently they agree, and how large your sample size needs to be. Hamel Husain has made a strong case for binary evaluations in production systems, they force sharper criteria and produce higher inter-rater agreement. Emily Reese and the Scale AI team showed that structured rubrics with explicit criteria (not just numeric labels) can improve model training by 28% over baseline Likert approaches.

Let's walk through the three main approaches and when each one actually works.

Check your understanding

Your team is building a customer support chatbot and wants to evaluate whether responses correctly escalate complex cases to human agents. Evaluators disagree frequently on 5-point ratings. What's the best next step?

Binary scales: clarity over nuance

Binary evaluations feel limiting at first, how can pass/fail capture the richness of AI output? But that constraint is the point. When you're forced to draw one line, you have to be specific about where it goes and why. Does the summary include all key facts? Yes or no. Does the code compile and produce correct output? Yes or no.

From what I've observed, binary rubrics have three practical advantages: they produce higher inter-rater agreement (often near 0.8 on Krippendorff's alpha), they require smaller sample sizes to detect real differences between model versions, and they push you to articulate concrete behavioral anchors rather than hiding behind vague labels like 'adequate' or 'good.'

The catch? Binary only works when you can define a meaningful threshold. For dimensions like factual correctness or policy compliance, that's usually straightforward. For fuzzier qualities, like tone or helpfulness, you might need more levels.

When binary breaks down

Binary scales work best for verifiable criteria with clear thresholds. They struggle when you genuinely need to distinguish gradations, like empathy in customer support (cold vs. professional vs. warm) or optimization in code (works but slow vs. reasonably efficient vs. highly optimized). If you find yourself wanting to mark something as 'partial pass,' that's a signal you need at least three levels.

Check your understanding

Likert and custom scales: when you need gradations

Okay, so when do you actually need a multi-point scale? When there are real, distinguishable levels of performance that matter to your use case. Meta's LLaMA-2 evaluation used separate rubrics for helpfulness, truthfulness, and harmlessness, each with multiple levels, because those dimensions genuinely vary in degree, not just presence/absence.

The key is behavioral anchors for every level. A 5-point helpfulness scale is useless if the levels are just labeled 1-5. But if you define them as: '1 = ignores the question entirely, 2 = addresses surface-level query but misses key context, 3 = answers core question but omits relevant follow-up, 4 = thorough response with minor gaps, 5 = comprehensive answer anticipating follow-up needs', now you've got something evaluators can apply consistently.

I've seen custom scales work well when you have domain-specific gradations. The Fitbit Insights Explorer rubric used a SHARP framework (Safety, Helpfulness, Accuracy, Relevance, Personalization) with tailored levels for health and wellness content, validated by both generalists and clinical specialists. The structure matched the actual performance spectrum they observed.

Start with 3 levels, expand only if needed

When you're unsure, default to a 3-point scale: below threshold, meets threshold, exceeds threshold. This forces you to define what 'good enough' looks like while giving space for excellence. If evaluators consistently struggle to distinguish between adjacent levels or you see central tendency bias, collapse back to binary. If you see meaningful clusters within a level, consider adding one more anchor point, but no more than 5 total.

Check your understanding

Match each evaluation scenario to the most appropriate scoring scale:

Key takeaways

Binary scales force sharper criteria, produce higher agreement, and require smaller samples, default to them when you can draw a clear threshold.
Multi-point scales (3-5 levels) work when there are real, distinguishable gradations of performance that matter to your use case.
Every scale level needs behavioral anchors, concrete descriptions of what that performance looks like, not just numeric labels.
More granularity is not always better: 7+ point scales usually just spread disagreement across more labels without adding signal.
Start with the simplest scale that captures meaningful differences, then expand only if evaluators consistently see clusters you're missing.

Your product check-in

Apply “Binary vs. Likert vs. Custom Scales: When to Use Each” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?