When to Run Evals: Development, Pre-Release, and Production Monitoring

Evals aren't a one-time report, they're a living test suite

Here's what I see all the time: teams build an AI feature, run some test cases before launch, breathe a sigh of relief when they get decent scores, and call it done. A month later they're scrambling to explain why production is a mess. The problem isn't that they ran evals, it's that they ran them once, at the wrong time.

Evals need to run at three distinct stages: development (fast feedback loops while you iterate), pre-release (quality gates before you ship), and production monitoring (catching drift and real failures after launch). Each stage has a different purpose, different speed requirements, and different tolerance for failure. Kevin Weil at OpenAI has been saying that writing evals is becoming a core PM skill, and he's right, because owning the quality bar means owning when and how you measure it.

Think of it like fitness tracking. You don't just weigh yourself once before a race and assume you're good to go. You check daily progress during training (development), do a final systems check before the event (pre-release), and monitor recovery and performance trends afterward (production). Miss any of those stages and you're flying blind.

Check your understanding

Your team just launched an AI chatbot that helps users find recipes. Two weeks later, users start complaining about weird suggestions (like 'chocolate cake' for 'vegan dinner'). During the post-mortem, you realize no one was checking production performance after launch. Which stage of evaluation did the team skip?

Development evals: Fast feedback loops that make iteration safe

When you're iterating on prompts, tweaking model settings, or experimenting with new retrieval logic, you need to know within seconds whether your change made things better or worse. That's what development evals do, they give you a tight feedback loop so you can try 10 versions of a prompt in an hour instead of guessing and hoping.

The trick here is speed over completeness. You're not trying to catch every edge case, you're trying to avoid breaking the main workflows. I've seen teams at companies like Descript evolve their approach: they keep a small suite of regression tests (maybe 20-50 examples) that cover happy paths and common user tasks. Run them in under a minute. If you break something obvious, you know immediately.

From what I've observed, the best development evals are the ones you actually run. That means fast, cheap, and easy to interpret, code-based checks for structured outputs, a handful of LLM-judge prompts for tone or relevance, and maybe some manual spot-checks when you're unsure. Hamel Husain talks a lot about this: move beyond vibe-checking by making measurement so cheap that it's easier to run an eval than to guess.

Start small with regression tests

Pre-release evals: Quality gates that decide whether you ship

This is where evals become gatekeepers. Pre-release evals run in CI/CD pipelines and use threshold gating, if your accuracy drops below 85%, deployment blocks automatically. No exceptions, no 'we'll fix it later.' It's the moment you enforce the quality bar you defined.

The difference from development is completeness and rigor. You're now testing happy paths, edge cases (long context, ambiguous queries), and known failure modes (hallucinations, tool misuse). You might run a larger test set, hundreds of examples instead of dozens, and you care about coverage across user segments, languages, or use cases. Companies like Duolingo run six-step eval processes here: human expert review, fairness checks for bias, accuracy verification. It's methodical.

One thing I've learned: pre-release evals only work if you actually respect the threshold. I've seen too many teams set a quality bar, watch it fail, then override it because 'we have a deadline.' That's how you end up like Microsoft Tay in 2016, shipping without adequate adversarial testing and getting manipulated into offensive outputs within hours. The eval was there to protect you. Let it.

Check your understanding

Match each evaluation stage to its defining characteristic.

Production monitoring: Catching what you couldn't predict

Here's the thing, you can't fully evaluate AI quality before launch. Real users will do things you never imagined. They'll paste in 10,000-word documents, ask questions in broken English, try to jailbreak your guardrails, or just use your product in ways that make zero sense to you. Production monitoring is how you learn what's actually happening.

This stage is about continuous measurement and alerting. You're tracking metrics like task success rate, user satisfaction scores, latency, and error patterns. You're sampling conversations for manual review (like Nurture Boss did with property management chats) and converting production failures into permanent test cases. You're watching for gradual drift, model updates, changing user behavior, or shifts in data distribution that slowly degrade quality.

Aman Khan at Arize AI talks about building a continuous improvement flywheel: production monitoring surfaces issues, you add those issues to your development eval suite, you fix them, you gate the fix with pre-release thresholds, and you monitor again. It's a loop, not a checklist. The companies that ship reliable AI products treat evals as infrastructure that runs forever, not a report you write once and forget.

Don't let production monitoring become vibe-checking

Check your understanding

Key takeaways

• Evals run at three stages: development (fast feedback during iteration), pre-release (quality gates before deployment), and production monitoring (catching drift and real failures post-launch).
• Development evals prioritize speed over completeness so you can iterate quickly without breaking core workflows.
• Pre-release evals use threshold gating to enforce your quality bar, if the eval fails, deployment blocks automatically.
• Production monitoring is essential because real users expose edge cases, drift, and failure modes you couldn't anticipate before launch.
• Treat evals as living infrastructure that runs continuously, not as a one-time report or pre-launch checklist.

Your product check-in

Apply “When to Run Evals: Development, Pre-Release, and Production Monitoring” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?