Turn user promises into test cases

A demo is one success; an evaluation tests a promise

Your assistant worked once. Now test whether it consistently respects the budget, preserves corrections, excludes unavailable dishes, and waits for approval. Each promise needs a case that could expose a failure.

A golden dataset is a curated collection of inputs and expected outcomes. It need not start large. Include ordinary requests, edge cases, and failures. Keep the expected outcome separate from what the current implementation happens to do.

A checklist and trace reveal which assistant behavior needs improvement. — Test the outcome, trace the failure, and improve one thing.

Check your understanding

Which is the strongest budget test?

Specify outcomes rather than one exact sentence

For a no-result case, acceptable behavior is an empty shortlist with an explanation. It need not use one fixed sentence. For totals, exact arithmetic is appropriate. Choose the comparison based on the behavior.

The Evaluation Lab runs executable checks on the same core used by retrieval and workflow. It reports each case and an aggregate pass count. These are deterministic component checks, not evidence that an unconnected language model is reliable.

Check your understanding

Match each case to the promise it tests.

Write cases before editing the solution

Create a $25 strict-boundary case, a vegetarian filter case, and an unapproved-submission case. Predict each result before running the lab. If the implementation fails, preserve the test while investigating the cause.

Anthropic’s evaluation guide emphasizes evaluating agent behavior beyond a single plausible final answer. Our small project follows that principle with explicit outcomes and traces.

Check your understanding

Write an expected result for a search where nothing qualifies.

Build checkpoint

Open the Evaluation Lab and follow the practice above. Each lab explains its inputs, its actual logic, and its limits. Save your observations below; your check-ins and reflection remain in Product Academy.

For an AI-assisted code change, download the lab bundle from the lab page. Ask your coding assistant to explain the relevant function first, change one behavior, and add one test that could fail. Compare the result with your prediction.

Save your experiment and check-in

Record your prediction, what you tried, what happened, and the evidence for your conclusion. Name one thing you can now do independently and one uncertainty to revisit.

Key takeaways

Convert user promises into falsifiable cases.
Use expected outcomes independent of the implementation.
State what the test suite does and does not cover.