Why Traditional QA Fails for AI Products

The deterministic assumption breaks down

Here's the thing, traditional QA works beautifully when you can write an assertion like assert output == 'Hello, World!'. The same input produces the same output, every single time. You run your test suite, everything goes green, you ship with confidence.

LLMs shatter that model. Ask an AI assistant to 'summarize this customer email' twice and you'll get two different summaries, both might be good, but they won't match character-for-character. AI systems produce distributions of outputs, not deterministic responses. From what I've seen, this is the single biggest thing that trips up teams coming from traditional software.

Kevin Weil at OpenAI put it directly: 'Writing evals is going to become a core skill for product managers.' The old playbook, unit tests with strict equality checks, just doesn't apply when your product's core behavior is probabilistic.

Check your understanding

You're building an AI feature that generates product descriptions from specs. A traditional QA engineer writes a test: 'Given input X, assert output exactly matches this string.' What's the main problem with this approach?

Real users expose what dev testing can't

Traditional QA assumes you can enumerate failure modes up front, write your test cases, run them, done. AI products fail in ways you can't predict until real users start poking at them. Remember Microsoft Tay in 2016? The chatbot passed internal testing, then users manipulated it into spewing offensive content within hours of launch.

From what I've observed, the nastiest AI failures happen at the edges: unusual phrasing, long context windows, ambiguous queries, adversarial inputs. You need coverage across happy paths, edge cases, and failure modes, and even then, production monitoring is essential because real-world usage will surface things your test suite missed.

Aman Khan at Arize AI talks about moving teams 'beyond vibe checks' to systematic quality management. That means evals running continuously: during development for fast feedback, in CI/CD as quality gates, and in production to catch drift and real-world failures post-launch.

Production is where the real test begins

Check your understanding

You need rubrics, not assertions

So if you can't write assert output == expected, what do you do? You switch from equality checks to tolerance-based evaluation. Does the summary capture the key points? Is the tone appropriate? Did it follow the constraints (like staying under budget, or citing sources)?

This is where code-based evals and LLM-as-judge come in. Code-based evals handle objective criteria fast, checking JSON structure, verifying a link is valid, confirming a number falls in range. LLM-as-judge handles the subjective stuff at scale, relevance, tone, semantic accuracy. Hamel Husain and Shreya Shankar have trained thousands of PMs on this: you define 'good' in natural language rubrics, then validate those rubrics against human judgment.

Look at Descript's video editing agent. They evolved from manual grading to LLM-graders with three criteria: 'don't break things, do what I asked, do it well.' They calibrate periodically against humans and run separate suites for quality benchmarking versus regression testing. The rubric is the product, it's how you operationalize what quality means for your specific use case.

Check your understanding

Match each evaluation approach to the type of quality check it's best suited for:

PMs must own the quality bar

Key takeaways

• Traditional QA assumes deterministic outputs; LLMs produce distributions, so strict equality assertions fail even when quality is good.
• AI products fail in unpredictable ways at the edges, you need continuous evals across development, CI/CD, and production, not one-time testing.
• Evaluation shifts from assertions to rubrics: code-based evals for objective checks, LLM-as-judge for subjective quality at scale.
• Production monitoring is essential, real users will expose edge cases and drift that no pre-launch test suite can anticipate.
• As a PM, you own the quality bar: defining 'good' in natural language rubrics tied to real user needs and real product risk.

Your product check-in

Apply “Why Traditional QA Fails for AI Products” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?