Regression testing a prompt or model
Also called: did the change break anything, eval set as a test suite
Re-running a fixed set of realistic cases with known-good answers every time you change the prompt, the model, or the settings — so you can see whether the change quietly broke things that used to work.
Small prompt edits and model upgrades routinely improve one case while silently regressing others. Without a saved eval set to re-run, you're shipping changes on a hunch and finding the regressions in production.
Trying it once. A single successful test after a change tells you that case works; a regression run tells you the change didn't cost you the cases you already had.
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1U.S. National Institute of Standards and Technology · 2023-01-26
First-pass citations, limited to primary sources; a reviewer will broaden and verify these before this entry leaves draft.