The newest direction in distillation automates the entire loop — the teacher generates data, fine-tunes the student, evaluates it, and iterates — reducing human involvement but making the initial teacher choice even more consequential.

Automated distillation lets the teacher run the full loop of data generation, training, evaluation, and iteration with minimal human involvement, though the teacher choice becomes even more critical since it drives the entire self-running process. ✦ AI generated

Author · ByteByteGo Newsletter · 2026-08-05 · original ↗

The newest direction in distillation reduces the manual effort by automating the whole process. In this setup, the large model runs the full loop on its own. It generates training data, fine-tunes the student, evaluates the student against a held-out set of examples it also generates, and repeats the cycle, adjusting what it produces until the student stops improving. The human role shrinks to defining the task and the success criteria at the start, with a final check on real data at the end. Recent work in 2026 applied this to a detection task and found that it worked well, with one finding worth keeping in mind. The choice of teacher model had a large effect on the outcome. Different teachers, given the same loop and the same student, produced students of noticeably different quality. So automation removes manual effort while making the initial choice of teacher more consequential, since that choice now drives an entire self-running process rather than a single training pass.

Read full article ↗excerpt · fair-use quotation