Creating Eval Datasets: Sourcing, Labeling, and Versioning

Why Your Eval Dataset Makes or Breaks Everything

Here's the thing, you can have the most elegant rubric in the world, but if your eval dataset doesn't match what users actually throw at your product, you're testing the wrong thing. The dataset is where theory meets reality. I've seen teams spend weeks tuning prompts against a beautifully curated dataset of edge cases, only to discover their production traffic looks nothing like it.

From what I've observed, the best eval datasets come from four complementary sources: production logs (what users actually ask), human annotation (domain experts labeling quality), synthetic generation (programmatically creating edge cases), and domain-specific knowledge bases (grounding truth for factual queries). Teresa Torres documented this when building Product Talk's Interview Coach AI, she started with real user transcripts, then systematically added adversarial cases where the AI might hallucinate or get tone wrong.

The goal isn't perfection. It's representativeness. Your eval dataset should reflect the distribution of production queries, the common cases that make up 80% of traffic, plus the edge cases that cause 80% of your support tickets. (Quality beats quantity every time.)

Check your understanding

You're building an eval dataset for a customer support chatbot. Your dataset currently has 200 carefully written test cases covering common queries like password resets and order status. Users report the bot fails on complex returns involving multiple items. What's the best next step?

Labeling: Getting from Raw Data to Ground Truth

Once you've sourced a batch of queries, you need labels, the ground truth that tells you whether a response is good or not. This is where things get messy. What counts as 'good' is often subjective, especially for open-ended tasks like summarization or creative writing.

Hamel Husain's field guide emphasizes something crucial: measure inter-annotator agreement using metrics like Cohen's κ (kappa) or Krippendorff's α. If two human raters disagree 40% of the time on what makes a good response, you don't have a labeling problem, you have a rubric problem. Go back and tighten your criteria until agreement is above 80%. (This is why we spent the last module building explicit rubrics.)

For specialized domains, LLM-as-judge hits a wall. Aman Khan's research with Arize AI shows that while strong LLM judges reach 80-90% agreement with humans on general tasks, that drops to 60-70% on expert domains like medical advice or legal reasoning. Use LLM judges for scalable initial screening, then route uncertain or high-stakes cases to human experts. It's a workflow, not an either-or.

Inter-annotator agreement is your canary

Check your understanding

Match each labeling scenario to the most appropriate evaluation approach:

Versioning: Treating Your Dataset Like Production Code

Let's get real, your dataset will change. User behavior shifts, your product adds features, edge cases emerge from production. If you're not versioning your eval dataset, you have no idea whether a score drop means your model got worse or your dataset got harder.

Implement version control the same way you would for code. Tag each dataset version with a date and change summary. When you add 50 new edge cases for multi-item returns, that's v1.2. When you remove 30 obsolete queries because you deprecated a feature, that's v1.3. Track scores against each version so you can compare apples to apples.

From what I've seen, successful teams schedule quarterly dataset reviews. They sample recent production logs, check if the eval distribution still matches reality, and prune test cases that no longer matter. Andrej Karpathy has talked about this in the context of Tesla's self-driving evals, the dataset has to evolve as the product and real-world conditions change, otherwise you're optimizing for yesterday's problems.

Trip-planning agent dataset evolution

Check your understanding

Key takeaways

• Your eval dataset must reflect production distribution, source from real logs, add edge cases, and prune obsolete queries regularly.
• Measure inter-annotator agreement (Cohen's κ or Krippendorff's α above 80%) to ensure your labeling rubric is clear enough.
• Use LLM-as-judge for scalable screening on general tasks, but route specialized or high-stakes decisions to human experts.
• Version your datasets like code, tag changes, track scores per version, and schedule quarterly reviews to keep evals relevant.

Your product check-in

Apply “Creating Eval Datasets: Sourcing, Labeling, and Versioning” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?