Article · 2026-08-05 · 6 moments

How Big Models Teach Small Models to Be Smart

In this article, we will walk through the idea from the ground up. ✦ AI generated

Data◆

DeepSeek demonstrated in early 2025 that a 7-billion-parameter student distilled from a large reasoning model outperformed a 32-billion-parameter model on competition mathematics, but these wins are narrow — distilled models still trail larger ones on broad general knowledge.

DeepSeek distilled models down to 1.5B-70B parameters, with a 7B student beating a 32B model on math benchmarks. However, these impressive results are confined to narrow, well-defined tasks like math and code, not general knowledge.

transcript

Author: A clear example came in early 2025 from a lab called DeepSeek. It used a large reasoning model to generate a set of training examples, then fine-tuned several existing smaller models on those examples. One result stood out. A 7-billion-parameter student scored higher than a 32-billion-parameter model on a competition mathematics benchmark, even though it was produced by plain fine-tuning on the larger model's outputs. The released family of distilled models ran from 1.5 billion parameters up to 70 billion, and the smaller ones were compact enough to run on a single graphics card, which is part of why the release drew so much attention. The practical effect was that strong performance on these narrow tasks became something a small team could run locally and cheaply, rather than only through a large hosted model. The qualifier matters as much as the headline. These wins tend to appear on narrow, well-defined tasks such as mathematics and code. On those tasks, a small distilled model can perform at a level its size would not suggest. Across broader measures of general knowledge, the same small models still trail the larger ones. For example, a model can become excellent at competition mathematics through distillation while remaining weaker at wide-ranging questions about the world. Therefore, a claim that a small model beats a large one is usually true in a specific, narrow sense.

gives example · 1

Mechanism

Distillation comes in three main forms — output distillation, feature distillation, and synthetic data distillation — and synthetic data distillation has become the most common because it only requires text output from the teacher.

Output distillation matches final soft labels, feature distillation matches internal representations, and synthetic data distillation uses teacher-generated datasets. The third form dominates because many strong models only expose text output, keeping probabilities and internals private.

transcript

Author: Distillation comes in three main forms, and they differ in what the student copies: Output distillation: The student matches the teacher's final outputs, including the soft labels described above. This is the original form from 2015 and the most direct one. Feature distillation: The student matches the teacher's internal representations, meaning the intermediate values a model computes while processing an input, before it settles on a final answer. The aim is a similar internal picture, not only a similar output. Google's EmbeddingGemma is trained this way, learning to produce internal representations close to those of a larger Gemini model. Synthetic data distillation: The teacher generates a dataset of examples, and the student is fine-tuned on that dataset the same way it would be trained on any ordinary data. Stanford's Alpaca was an early case, fine-tuned on examples produced by an existing large model to improve how well it followed instructions. The third form has become the most common approach in practice, and part of the reason comes down to access. Many strong models are reachable only through an interface that returns text, with their internal values and probabilities kept private. When those internals are out of reach, generating data is the route that still works.

extends · 1

Claim

Distillation has four key limits: a ceiling effect from the teacher, degradation when the teacher-student capacity gap is too wide, architecture mattering more than size, and the teacher passing on unintended behavioral traits through hidden signals in data.

The teacher sets a quality ceiling, too large a capacity gap hurts transfer, a better base architecture can beat a larger one, and a 2025 Nature study showed behavioral traits can transfer through distillation even when data is filtered.

transcript

Author: Distillation has clear limits, and they matter when deciding whether it fits a given problem. A ceiling effect from the teacher: A student trained on a teacher's output tends to stay at or below the teacher's level on the kind of data they saw. When the teacher produces a wrong answer, the student learns that wrong answer along with the right ones. The teacher's quality sets the bar, which makes the choice of teacher one of the most consequential decisions in the process. A wider gap can hurt: A larger, stronger teacher does not always produce a better student. When the gap between teacher and student is very wide, transfer can degrade, because the student has too little capacity to absorb everything that a much larger model expresses. Research on this capacity gap has found that the strongest available teacher is sometimes a poor choice. A set of methods exists to bridge wide gaps by adding a middle step, where the teacher trains a mid-sized model and that model trains the small student, so each handoff spans a smaller distance. Architecture can outweigh size: The design of the base model can matter more than its parameter count. In one study, a 32-billion-parameter student outperformed a 70-billion-parameter student on the same task, because the smaller one was built on a stronger base architecture. Size alone is a weak predictor of how well distillation will go. The teacher can pass on more than the task: In a 2025 study later published in Nature, a teacher model with a particular trait, a tendency to favor owls, was used to generate training data made up only of number sequences. A student trained on those numbers picked up the same preference for owls, even after the data was filtered to remove any visible trace of the trait. The same effect appeared with more serious behaviors, and it occurred only when the teacher and student shared the same base model. The takeaway is that distillation can carry across more than the task being taught, and that filtering the visible data is sometimes too coarse to stop it.

Definition

Knowledge distillation trains a separate, smaller model to copy the behavior of a larger one, producing a genuinely new model rather than a compressed version of the original.

Distillation produces a separate model with its own parameters, unlike compression methods like quantization and pruning which shrink an existing model.

transcript

Author: Distillation trains a new, smaller model to copy the behavior of a larger one. The setup involves two models: The first is a large, capable model called the teacher. The second is a smaller model called the student, which is trained to reproduce the teacher's outputs. Once training finishes, the student runs on its own, and the teacher steps out of the picture. A common assumption is that the student is the teacher in compressed form. The reality, however, is different. Compression methods such as quantization and pruning start with one model and reduce its footprint by storing its numbers at lower precision or removing parts that contribute little to the result. The model stays the same model, smaller and lighter. Distillation, on the other hand, produces a genuinely separate model, with its own parameters and often a different design, whose goal during training is to behave like the teacher. One operation shrinks an existing model. The other trains a fresh one.

explains mechanism · 1

Mechanism◆

Soft labels from a teacher model carry richer information than hard labels because they encode the full probability distribution across categories, which reveals relationships between classes.

Instead of a single correct answer (hard label), a teacher provides probabilities across all options (soft labels), which researchers call dark knowledge — this reveals how categories relate to each other and provides a stronger training signal.

transcript

Author: Learning from a model's output beats learning from raw data because the output carries more information than a plain answer. Standard training data gives one answer per example. An image of a cat carries the label 'cat,' and the model is rewarded for producing 'cat' and penalized for anything else. A teacher model offers something richer. Instead of a single answer, its output is a set of probabilities across the options, such as cat at 0.70, dog at 0.25, and fox at 0.05. That full set of probabilities is called a soft label, in contrast to the single hard label found in ordinary data. The extra numbers carry additional information. They show that the teacher's output ranks dog as a plausible alternative and fox as a distant one, which says something about how the categories relate to each other. Researchers sometimes call this dark knowledge, meaning the structure hidden in a model's confidence that a bare label leaves out. During training, the student works to match this distribution. It is scored on how far its own probabilities sit from the teacher's, and training pushes it to close that gap. In other words, the student learns the teacher's whole pattern of confidence rather than a single right answer, and that pattern is a stronger training signal than a one-word label. This is the core reason distillation works as well as it does. A single correct label discards the relationships between options, and soft labels keep them.

Prediction

The newest direction in distillation automates the entire loop — the teacher generates data, fine-tunes the student, evaluates it, and iterates — reducing human involvement but making the initial teacher choice even more consequential.

Automated distillation lets the teacher run the full loop of data generation, training, evaluation, and iteration with minimal human involvement, though the teacher choice becomes even more critical since it drives the entire self-running process.

transcript

Author: The newest direction in distillation reduces the manual effort by automating the whole process. In this setup, the large model runs the full loop on its own. It generates training data, fine-tunes the student, evaluates the student against a held-out set of examples it also generates, and repeats the cycle, adjusting what it produces until the student stops improving. The human role shrinks to defining the task and the success criteria at the start, with a final check on real data at the end. Recent work in 2026 applied this to a detection task and found that it worked well, with one finding worth keeping in mind. The choice of teacher model had a large effect on the outcome. Different teachers, given the same loop and the same student, produced students of noticeably different quality. So automation removes manual effort while making the initial choice of teacher more consequential, since that choice now drives an entire self-running process rather than a single training pass.

extends · 1

Highlight slides

Hard Labels Discard Information✦ from: Soft labels from a teacher model carry richer information than hard labels because they encode the full probability distribution across categories, which reveals relationships between classes.Small model beats large, on math✦ from: DeepSeek demonstrated in early 2025 that a 7-billion-parameter student distilled from a large reasoning model outperformed a 32-billion-parameter model on competition mathematics, but these wins are narrow — distilled models still trail larger ones on broad general knowledge.Soft Labels Carry Dark Knowledge✦ from: Soft labels from a teacher model carry richer information than hard labels because they encode the full probability distribution across categories, which reveals relationships between classes.Wins are narrow, not general✦ from: DeepSeek demonstrated in early 2025 that a 7-billion-parameter student distilled from a large reasoning model outperformed a 32-billion-parameter model on competition mathematics, but these wins are narrow — distilled models still trail larger ones on broad general knowledge.

Related episodes

Article · Import AI

Import AI 464: Fables writes GPU kernels; AI automation; and analog computation

→

Video · The Cognitive Revolution

Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

→

Article · Import AI

Import AI 465: Open vs closed gaps; Kimi K3; Demis' big policy plan

→