TOPIC 05 OF 06
Watch it in production
Logging and human review after launchTesting before launch isn't the end — real inputs drift from your test set, and quality can decay quietly. Observability means recording what the system received, did, and produced, so you can understand behaviour after the fact. Paired with regularly sampling real outputs and having a person rate them, it's how you catch new failure modes and 'technically passing but unhelpful' answers before users make you aware of them.
A live assistant starts getting a question type it never saw in testing. Because outputs are logged and sampled weekly, a reviewer spots the poor answers early — instead of learning about them from a spike in complaints months later.
Passive monitoring — a dashboard nobody reads — isn't review. And logging everything forever is its own privacy problem: be deliberate about what you keep and for how long, balancing the need to explain behaviour against the cost of retaining user content.
For one live AI feature, ask two questions: can we see what it actually did last week, and does anyone read a sample of it? If not, that's the first thing to add.