Decide whether an improvement is ready to ship
A better average can hide a worse product
Imagine a change makes most answers faster but occasionally removes a dietary constraint. Report quality by important case categories, not just one average. A change is acceptable only if it satisfies the critical requirements.
Track task success, constraint violations, recovery behavior, latency, and cost separately. For live systems, percentile latency can reveal slow experiences hidden by the mean. The local lab timing is component execution time, not network or model latency.
Check your understanding
Nine tests pass and the approval gate fails. What should your decision be?
Compare a baseline with a controlled change
The Evaluation Lab offers baseline, omitted-fee bug, and missing-approval bug variants. The same cases run against each variant. Your job is to explain which promise each failure breaks.
A small fixture suite is a starting point, not proof of universal safety. Define additional representative requests and repeated live-model trials before relying on stochastic behavior. A passing test can only cover what it actually exercised.
Check your understanding
Write a release decision with a rollback condition
For this prototype, require all critical fixture checks to pass and document remaining live-integration gaps. In a real rollout, start small, monitor the same important behaviors, and define a condition that pauses or rolls back the release.
Your module artifact is an evaluation report: baseline results, one intentionally broken variant, root cause, and a release decision. Export the lab report and save your interpretation in Product Academy.
Check your understanding
State a rollout stop condition for the dinner assistant.
Build checkpoint
Open the Evaluation Lab and follow the practice above. Each lab explains its inputs, its actual logic, and its limits. Save your observations below; your check-ins and reflection remain in Product Academy.
For an AI-assisted code change, download the lab bundle from the lab page. Ask your coding assistant to explain the relevant function first, change one behavior, and add one test that could fail. Compare the result with your prediction.
Save your experiment and check-in
Record your prediction, what you tried, what happened, and the evidence for your conclusion. Name one thing you can now do independently and one uncertainty to revisit.
Key takeaways
- Separate critical gates from average quality.
- Compare changes on the same test set.
- Record untested gaps and an observable stop condition.