← Concept IndexDEFINITION WHY IT MATTERS COMMONLY CONFUSED WITH
Inference cost and latency
Also called: token cost, response time, throughput
Every call to a model costs money (usually per token) and takes time. At real volume, the size of the model, the length of the prompt, and how often you call it decide both your bill and how fast the feature feels.
A demo that calls the biggest model on every keystroke is fine for one person and ruinous for a million. Choosing a smaller model where it suffices, trimming context, and caching repeated work is often the difference between a feature that ships and one that can't afford to run.
Model quality. The most capable model isn't automatically the right one in production — the right one is the cheapest and fastest that still passes your evaluation for the task.