TOPIC 04 OF 06
Catch regressions
Did this change break anything?Every time you change the prompt, swap the model, or adjust a setting, you can improve one case while silently breaking others. Re-running your fixed test set after each change — a regression run — is what turns 'I think this is better' into 'I can see it's better on nine of ten cases and worse on none.' Without it, you ship changes on a hunch and find the breakage in production.
You tweak a prompt to fix one awkward output. It works — but your regression run shows two cases that used to pass now fail. You caught it in seconds instead of in a customer complaint.
Trying the change once on the case you were fixing tells you that case works now. Only re-running the whole set tells you the change didn't cost you ground you already had.
Before your next prompt or model change, save the current outputs on your test set. After the change, compare. That's a regression run.