Distillation comes in three main forms — output distillation, feature distillation, and synthetic data distillation — and synthetic data distillation has become the most common because it only requires text output from the teacher.

Output distillation matches final soft labels, feature distillation matches internal representations, and synthetic data distillation uses teacher-generated datasets. The third form dominates because many strong models only expose text output, keeping probabilities and internals private. ✦ AI generated

Author · ByteByteGo Newsletter · 2026-08-05 · original ↗

Distillation comes in three main forms, and they differ in what the student copies: Output distillation: The student matches the teacher's final outputs, including the soft labels described above. This is the original form from 2015 and the most direct one. Feature distillation: The student matches the teacher's internal representations, meaning the intermediate values a model computes while processing an input, before it settles on a final answer. The aim is a similar internal picture, not only a similar output. Google's EmbeddingGemma is trained this way, learning to produce internal representations close to those of a larger Gemini model. Synthetic data distillation: The teacher generates a dataset of examples, and the student is fine-tuned on that dataset the same way it would be trained on any ordinary data. Stanford's Alpaca was an early case, fine-tuned on examples produced by an existing large model to improve how well it followed instructions. The third form has become the most common approach in practice, and part of the reason comes down to access. Many strong models are reachable only through an interface that returns text, with their internal values and probabilities kept private. When those internals are out of reach, generating data is the route that still works.

Read full article ↗excerpt · fair-use quotation