DefinitionArticle
Knowledge distillation trains a separate, smaller model to copy the behavior of a larger one, producing a genuinely new model rather than a compressed version of the original.
Distillation produces a separate model with its own parameters, unlike compression methods like quantization and pruning which shrink an existing model. ✦ AI generated
Author · ByteByteGo Newsletter · 2026-08-05 · original ↗
Distillation trains a new, smaller model to copy the behavior of a larger one. The setup involves two models: The first is a large, capable model called the teacher. The second is a smaller model called the student, which is trained to reproduce the teacher's outputs. Once training finishes, the student runs on its own, and the teacher steps out of the picture. A common assumption is that the student is the teacher in compressed form. The reality, however, is different. Compression methods such as quantization and pruning start with one model and reduce its footprint by storing its numbers at lower precision or removing parts that contribute little to the result. The model stays the same model, smaller and lighter. Distillation, on the other hand, produces a genuinely separate model, with its own parameters and often a different design, whose goal during training is to behave like the teacher. One operation shrinks an existing model. The other trains a fresh one.
Read full article ↗excerpt · fair-use quotation
Around this claim
This moment responds to
extends → Distillation comes in three main forms — output distillation, feature distillation, and synthetic data distillation — and synthetic data distillation has become the most common because it only requires text output from the teacher.Author · ByteByteGo Newslettergives example → DeepSeek demonstrated in early 2025 that a 7-billion-parameter student distilled from a large reasoning model outperformed a 32-billion-parameter model on competition mathematics, but these wins are narrow — distilled models still trail larger ones on broad general knowledge.Author · ByteByteGo Newsletterextends → The newest direction in distillation automates the entire loop — the teacher generates data, fine-tunes the student, evaluates it, and iterates — reducing human involvement but making the initial teacher choice even more consequential.Author · ByteByteGo Newsletter