Red-teaming
Also called: adversarial testing, trying to break it, stress-testing
Deliberately attacking your own AI system to find how it fails — probing for harmful outputs, prompt injection, data leakage, jailbreaks, and edge cases — before real users or bad actors find them.
Normal evaluation measures whether a system does the right thing on expected inputs; red-teaming measures what happens under hostile or unusual ones. For anything that acts, handles sensitive data, or faces the public, the failure you didn't go looking for is the one that ends up in the news.
Regular testing. A test suite confirms known-good behavior; red-teaming is an open-ended hunt for unknown-bad behavior, and it's never really 'done'.
- Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1U.S. National Institute of Standards and Technology · 2023-01-26
First-pass citations, limited to primary sources; a reviewer will broaden and verify these before this entry leaves draft.