← Concept IndexDEFINITION WHY IT MATTERS COMMONLY CONFUSED WITH
Distillation
Also called: teacher-student models, compressing a model
Training a smaller, cheaper 'student' model to mimic the behaviour of a larger 'teacher' model, capturing much of its capability at a fraction of the running cost.
It's a main reason small, fast models keep getting surprisingly good, and it's how a capable model can be made affordable to run at scale. It shows capability and size aren't rigidly linked.
Fine-tuning. Distillation is about transferring a big model's behaviour into a smaller one; fine-tuning adapts a given model to a specific task or style.