LANDMARK · 6 MIN · BUILD
Latency budgets
After this landmark, you can set a latency budget for an AI feature based on where it sits in the product, and choose an architecture that can meet it.
You can still explore it. We’re showing the shared explanation and a related practical view without hiding the knowledge.
Latency tolerance depends entirely on context: a background report generation job can take minutes; a chat response feels broken past a few seconds; an autocomplete suggestion needs to arrive in well under a second or it’s useless. Setting the budget first (before choosing a model, whether to stream the response, or whether to add a retrieval step), determines what architecture is even viable, because retrieval, multi-step agent loops, and larger models all add latency that a tight budget may not tolerate.
Streaming partial output, using a smaller model for latency-sensitive paths, and pre-computing what can be pre-computed are the standard levers once the budget is set.
This is the canonical concept. It stays the same across learner lenses so personalization never changes the underlying facts.
What this looks like for you
You feel this directly as the difference between an AI feature that feels instant and one that feels sluggish, the sluggish ones usually chained several slow steps (search, then a large model call) without streaming anything back to you in between.
MAKE A DECISION
A team is building a live coding-assistant autocomplete feature, expected to respond as the user types. Which architecture fits the latency requirement?
CARRY THISFor one AI feature you build or use, name its actual latency budget based on where it sits in the UI. Does the current architecture (model size, retrieval steps, agent loop) fit that budget, or exceed it?