Coverage Strategy: Happy Paths, Edge Cases, and Failure Modes
Why your happy-path tests aren't enough
Here's the thing, most teams start by testing the obvious, mainstream scenarios. A travel planning AI gets tested with "relaxing weekend getaway near San Francisco under $1,000" and if it returns flights, hotels, and a reasonable itinerary, everyone feels good. But from what I've seen, production failures live in the gaps between these comfortable cases.
Traditional software QA taught us to write tests for the expected workflows. AI products break that assumption. The distribution of user inputs is vastly wider, the outputs are probabilistic, and the failure modes aren't crashes, they're subtle hallucinations, tone misfires, or tool misuse that slip through deterministic checks. Microsoft's Tay chatbot (2016) is the cautionary tale here: it passed all the happy-path tests, then got manipulated by users to produce offensive content within hours of launch.
A coverage strategy is your map of what to test and why. It breaks down into three zones: happy paths (the standard workflows you design for), edge cases (unusual but valid inputs your system should handle gracefully), and failure modes (AI-specific ways things go wrong that you need to detect and mitigate). You need all three, and you need them running continuously.
Check your understanding
Your AI email assistant successfully drafts replies to common customer inquiries about pricing and returns. Before launch, which additional test category would most likely catch production issues users will encounter?
Mapping your edge cases before they find you
Edge cases aren't exotic, they're just the long tail of real usage. A few I've seen trip up teams: queries with typos or slang, requests in the wrong language, inputs that are way longer than your test examples, ambiguous phrasing that could mean two things, and requests that your AI should politely refuse but might try to answer anyway.
The trick is to systematically enumerate what 'unusual' means for your product. Duolingo's English Test does this rigorously, their six-step eval process includes fairness checks (testing across demographics) and validity reviews (edge cases that probe whether the test measures what it claims to). For a coding assistant, edge cases might include incomplete code snippets, requests for deprecated libraries, or prompts that ask for help with malicious scripts.
Start by listing the dimensions of variation in your inputs: length (very short, very long), clarity (crystal clear vs. ambiguous), domain (in-scope vs. adjacent vs. completely off-topic), format (clean vs. messy), and user intent (genuine vs. exploratory vs. adversarial). Then pick a few representative test cases along each dimension. You're not trying to test every possible input, you're sampling the space intelligently.
Start with production logs
If you already have users, your best edge case library is sitting in your logs. Pull the queries that confused your AI, the ones that got thumbs-down, or the sessions where users rephrased three times. Those are your real edge cases, not the ones you imagine in a conference room.Check your understanding
Failure modes: the AI-specific ways things break
Failure modes are different from edge cases. Edge cases are unusual inputs; failure modes are the specific ways your AI misbehaves when stressed. Hamel Husain trains teams to do systematic error analysis, categorizing production failures into a taxonomy you can test against.
The big ones I see over and over: hallucinations (confidently generating false information), prompt injection (users manipulating your system prompt to bypass guardrails), tool misuse (calling the wrong API or passing malformed parameters in agentic systems), context window overflow (losing critical information when the conversation gets long), and tone drift (starting professional, ending snarky). Each of these needs explicit test cases.
Harvey, the legal AI company, built adversarial tests for source traceability, they found that general-purpose models (GPT-4o, Claude 3.5) scored 8-24% on providing accurate citations, while their specialized legal model hit 74%. That gap only showed up because they tested the specific failure mode they cared about: making up sources. Generic benchmarks would have missed it entirely.
Failure modes evolve with your product
When you add a new tool or capability to your AI agent, you unlock new failure modes. A travel planner that only recommended hotels had limited downside. Add flight booking, and now it can double-book, drain a user's budget, or pick flights with impossible layovers. Your eval suite needs to grow in lockstep with your feature set.Check your understanding
Imagine you're building an AI coding assistant that can read a user's codebase, answer questions, and suggest refactorings. List three specific failure modes you would test for and explain why each matters for user trust.
Key takeaways
- A coverage strategy maps three zones: happy paths (standard workflows), edge cases (unusual but valid inputs), and failure modes (AI-specific ways things break).
- Edge cases aren't exotic, they're the long tail of real usage, and production logs are your best source for finding them.
- Failure modes like hallucinations, prompt injection, and tool misuse need explicit adversarial tests; they won't show up in happy-path evaluation.
- Your eval suite must evolve as your product does, each new capability unlocks new failure modes to test.
- Generic benchmarks miss product-specific risks; you must define and test the failure modes that matter for your users.
Your product check-in
Apply “Coverage Strategy: Happy Paths, Edge Cases, and Failure Modes” to a product or workflow you know. What would you try, what could go wrong, and what evidence would help you decide?