Reading Eval Results: Aggregate Scores, Distributions, and Regressions

Why Averages Lie to You

Here's the thing, when you first run an eval, you'll see a single number at the top of the dashboard: average score 7.2/10. Your instinct is to declare victory or sound the alarm. From what I've seen, that's exactly where most PMs go wrong.

The mean score hides the shape of what's happening. Maybe 80% of your responses are perfect 9s and 10s, but 20% are complete failures scoring 1 or 2. That averages to a respectable 7.6, but you've shipped a product that catastrophically fails one in five times. Or maybe everything clusters around 7, mediocre across the board, no disasters but nothing great either. Same average, completely different product reality.

What you actually need is the distribution. Look at the histogram. Where do scores cluster? How fat is the tail of terrible responses? Are there bimodal peaks suggesting the model handles some query types beautifully and others not at all? (I've seen teams ship features based on a good average score, only to discover in production that a specific user segment was getting systematically bad responses the whole time.)

Check your understanding

Your travel booking agent scores an average of 7.8/10 on helpfulness. When you examine the distribution, you find 70% of responses score 9-10, and 30% score 3-4. What does this pattern suggest about your product readiness?

Three Levels of Reading: Mean, Distribution, Examples

I think of eval results as having three layers you need to read through, each one gets you closer to what's actually broken (or working).

First layer: mean scores per criterion. Not just overall average, but broken down by each rubric dimension. Your chatbot might score 8.5 on tone but 5.2 on factual accuracy. That tells you where to focus. Second layer: score distributions and patterns. Group results by query type, user segment, or response length. Does the model fail consistently on multi-step reasoning? Does it excel on product questions but fumble on policy edge cases? These patterns point to dataset gaps or prompt weaknesses.

Third layer, and this is where the real insight lives, worst-performing examples. Sort by lowest score and read the actual responses. You'll spot patterns the aggregate numbers can't show you: the model invents product features that don't exist, it switches tone mid-response, it misunderstands negation in questions. Teresa Torres documented this process for Product Talk's Interview Coach, the systematic failures she found by reading low-scoring examples drove her entire next iteration of prompts and knowledge base structure.

Start with the bottom ten

When you first look at eval results, ignore the summary stats. Sort by score ascending and read the ten worst responses. You'll learn more in ten minutes than an hour of staring at dashboards. Look for what the failures have in common, that's your next sprint.

Check your understanding

Strong LLM judges achieve 80-90% agreement with humans on general tasks, but only ___% agreement on expert domains requiring specialized knowledge.

Detecting Regressions: Comparing Against Baselines

Once you've shipped and your eval pipeline is running continuously, the question shifts from 'is this good?' to 'did this get worse?' That's regression detection, and it's where evals earn their keep in production.

You're watching for two kinds of drift. Prompt drift happens when small wording changes accumulate, you tweak the system message to handle edge case A, then adjust it for edge case B, and three months later nobody remembers why it says what it says and performance has quietly degraded. Model drift happens when your upstream provider (OpenAI, Anthropic, whoever) updates their model and your prompts no longer work as well as they used to.

The way you catch both is historical baselines. Run the same eval dataset every week (or after every significant change). Plot the scores over time per criterion. When you see a drop, even a small one, say from 8.2 to 7.8, that's your signal to investigate. Hamel Husain's Field Guide emphasizes treating evals like regression tests: any drop below your minimum acceptable threshold should block deployment until you understand why.

Check your understanding

Non-deterministic systems need multiple runs

Unlike traditional software tests, AI systems with temperature > 0 produce different outputs for identical inputs. To get stable eval results, either set temperature to zero for reproducibility, or aggregate scores across multiple runs and use confidence intervals to quantify uncertainty. Make sure your sample size is large enough to detect meaningful differences, 20 examples won't cut it.

Key takeaways

Mean scores hide the story, always examine the distribution, look for bimodal clusters, and identify fat tails of terrible responses.
Read eval results in three layers: mean scores per criterion, patterns across query types or segments, and the actual worst-performing examples.
Regression detection requires historical baselines, run evals continuously to catch prompt drift and model drift before they degrade user experience.
LLM-as-judge works for scalable screening (80-90% human agreement on general tasks) but drops to 60-70% in expert domains, so pair automation with targeted human review.
Non-deterministic systems need multiple runs or temperature zero for reproducibility, plus confidence intervals to know if a score change is real or noise.

Your product check-in

Apply “Reading Eval Results: Aggregate Scores, Distributions, and Regressions” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?