TOPIC 02 OF 06
Build a small test set
Realistic cases, known-good answersAn evaluation is a fixed set of realistic inputs paired with the answer you'd accept as good. It doesn't need to be big — a dozen well-chosen cases that cover your normal work and your nasty edge cases will catch far more than trying random prompts. The value is that it's fixed: you can run the same cases again after any change and compare.
For an email-triage assistant: collect ten real (anonymised) emails — some urgent, some spam, some ambiguous — and write the correct triage for each. That's your test set. Now 'is it good?' has a concrete, repeatable answer.
A test set of only easy cases flatters the system. Deliberately include the edge cases you're worried about — the confusing, the adversarial, the rare-but-costly — because those are where it will actually fail.
Write down five realistic cases for one AI task, each with the answer you'd want. You now have the start of a real eval.