LANDMARK · 7 MIN · UNDERSTAND
Size, distillation, and quantisation
After this landmark, you can explain what quantisation and distillation trade away, and choose a model size on evidence rather than on instinct.
↗
This concept is shared, but the Everyday lens is less central here.
You can still explore it. We’re showing the shared explanation and a related practical view without hiding the knowledge.
A model’s parameters are numbers that have to be stored and moved, so size determines memory, cost, and speed before it determines anything about quality. Two techniques shrink that footprint. Quantisation stores the same parameters at lower numeric precision — dropping from 16-bit to 8- or 4-bit roughly halves or quarters the memory — with a modest quality cost that is often barely visible on everyday tasks and more noticeable on hard reasoning. Distillation is different: a large “teacher” model trains a much smaller “student” to imitate its outputs, producing a genuinely smaller model that punches above its parameter count. Nearly every small model you can run locally is one or both of these. The practical upshot is that “which model?” is not a single-axis question. A distilled, quantised model that answers in 200 milliseconds on hardware you already own can be the right choice over a frontier model that is better on paper.
This is the canonical concept. It stays the same across learner lenses so personalization never changes the underlying facts.
◌ EVERYDAY LENS
What this looks like for you
This is why an assistant can run on your phone or laptop with no network. Local models are meaningfully weaker at hard reasoning but keep your data on the device, which for private material is often the trade worth making.
MAKE A DECISION
A pipeline classifies 50,000 support emails a day into eight categories, currently using a frontier model. What is the most sensible efficiency move?
CARRY THISTake one repeated AI task and run it on the smallest model you have access to. Note whether the output difference would be noticed by whoever consumes it.