Choosing Rubric Dimensions: Accuracy, Helpfulness, Safety, Tone
Why Generic Quality Scores Don't Cut It
Here's the thing, when most teams start evaluating AI outputs, they reach for a single 1-5 quality score. It feels simple. It feels measurable. But from what I've seen, it hides more than it reveals. A chatbot response can be factually perfect but sound like a robot, or warm and empathetic but completely wrong about your return policy.
Multi-dimensional rubrics separate what matters into distinct axes you can measure independently. Aman Khan, who developed the popular evals course with Andrew Ng, calls this essential to moving "beyond vibe checks", you need to know which dimension is broken, not just that something feels off. When Meta evaluated LLaMA-2, they didn't ask "Is this good?" They measured helpfulness, truthfulness, and harmlessness as separate dimensions, because a model can ace one and fail another.
The dimensions you pick depend entirely on your use case. A customer support bot cares about policy accuracy, escalation handling, and empathy. A code generator cares about correctness, efficiency, and readability. There's no universal set, task-specific rubrics outperform generic ones every time.
Check your understanding
You're evaluating a customer service chatbot. One response correctly cites the return policy but uses curt, robotic language that frustrated the user. Another response is warm and empathetic but gives the wrong refund deadline. Why does a multi-dimensional rubric help here more than a single 1-5 quality score?
The Four Core Dimensions (and When to Add More)
Most AI product rubrics start with four foundational dimensions: Accuracy (factual correctness, consistency with source material), Helpfulness (does it actually address what the user needs), Safety (bias, toxicity, harm), and Tone (style, empathy, formality). These cover the basics across a lot of use cases.
But you'll often need to add task-specific dimensions. Fitbit's Insights Explorer added Relevance and Personalization for health guidance, validated by both generalists and clinical specialists. Code generation rubrics often include Efficiency (runtime, memory) and Security (no SQL injection patterns, proper input validation). Summarization systems care about Coverage (all main points included) and Conciseness (no fluff).
The trick is to start with what you'll act on. If you're not going to change the model or prompt based on a dimension, don't measure it. I've seen teams add seven dimensions because it felt thorough, then realize they only ever look at two. Choose dimensions that map to decisions, which training data to add, which guardrails to tune, which user complaints to prioritize.
Start narrow, expand as you learn
Don't try to build the perfect comprehensive rubric on day one. Start with 2-3 dimensions that directly tie to your biggest user complaints or model failures. Add dimensions only when you have evidence they predict outcomes you care about, otherwise you're just collecting noise.Check your understanding
Match each AI application to the dimension that matters most for its specific use case:
Weighting What Matters: Not All Dimensions Are Equal
Once you've chosen your dimensions, you need to decide which ones matter most. This isn't philosophical, it's about matching your rubric to real consequences. In the Scale AI "Rubrics as Rewards" research, Emily Reese and her team found that semantic weighting (marking criteria as Essential vs. Important) led to 28% better model performance than treating everything equally.
For code generation, one common weighting is Correctness 50%, Efficiency 30%, Style 20%. Code that doesn't run is worthless, slow code annoys users, ugly code is a maintenance tax. The weights reflect that hierarchy. For customer support, you might weight Accuracy and Escalation heavily (a wrong answer or missed handoff is costly) and Tone lower (nice to have, but not mission-critical if the answer is right).
From what I've observed, the weighting conversation forces clarity. If your team can't agree whether Safety or Helpfulness matters more for your chatbot, you probably don't have alignment on what success looks like. Use the rubric design process to surface and resolve that, before you ship.
Check your understanding
Rubrics reveal misalignment fast
One of the most valuable side effects of rubric design is discovering where your team disagrees about priorities. If engineering wants to weight Efficiency at 50% and your design lead insists Tone is mission-critical, you've uncovered a strategic gap that no amount of evaluation data will fix. Resolve it now, not after launch.Key takeaways
- Single quality scores hide trade-offs; multi-dimensional rubrics show you exactly which aspect of your AI output is broken.
- The four foundational dimensions, Accuracy, Helpfulness, Safety, Tone, cover many use cases, but task-specific dimensions (Efficiency, Personalization, Coverage) often matter more.
- Weight dimensions to match real consequences: a wrong answer in customer support is worse than an awkward tone.
- Choose only dimensions you'll act on; measuring seven things when you only ever fix two just creates noise.
- If your team can't agree on weighting, you don't have alignment on what success looks like, fix that before you ship.
Your product check-in
Apply “Choosing Rubric Dimensions: Accuracy, Helpfulness, Safety, Tone” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?