LANDMARK · 7 MIN · UNDERSTAND
What happens at inference
After this landmark, you can explain why the first word takes longer than the rest, and identify the levers that actually reduce latency and cost.
↗
This concept is shared, but the Everyday lens is less central here.
You can still explore it. We’re showing the shared explanation and a related practical view without hiding the knowledge.
Inference is the act of running a trained model on an input. It has two phases with different characteristics. Prefill reads your whole prompt at once and can be processed in parallel, so its cost scales with prompt length but it is comparatively efficient. Decode then produces the answer one token at a time, and each token requires another pass over the model, so output length drives latency almost linearly. This is why a long prompt with a short answer feels fast, and a short prompt asking for a long answer feels slow — and why input tokens are usually priced lower than output tokens. The other half of the picture is the attention cache: as generation proceeds, the model keeps intermediate state for every token so far, which grows with context length and consumes memory. Long conversations get slower and more expensive not because the model is tiring but because that state grows.
This is the canonical concept. It stays the same across learner lenses so personalization never changes the underlying facts.
◌ EVERYDAY LENS
What this looks like for you
Streaming is why text appears word by word — you’re watching decode happen. Asking for a shorter answer genuinely makes it faster, in a way that asking it to “be quick” does not.
MAKE A DECISION
An internal assistant sends the same 3,000-token instruction block with every request, then produces short answers. Users complain about cost. What is the first move?
CARRY THISFor one AI feature you use, note the typical input length and output length. Work out which of the two is driving its cost.