Soft labels from a teacher model carry richer information than hard labels because they encode the full probability distribution across categories, which reveals relationships between classes.

Learning from a model's output beats learning from raw data because the output carries more information than a plain answer. Standard training data gives one answer per example. An image of a cat carries the label 'cat,' and the model is rewarded for producing 'cat' and penalized for anything else. A teacher model offers something richer. Instead of a single answer, its output is a set of probabilities across the options, such as cat at 0.70, dog at 0.25, and fox at 0.05. That full set of probabilities is called a soft label, in contrast to the single hard label found in ordinary data. The extra numbers carry additional information. They show that the teacher's output ranks dog as a plausible alternative and fox as a distant one, which says something about how the categories relate to each other. Researchers sometimes call this dark knowledge, meaning the structure hidden in a model's confidence that a bare label leaves out. During training, the student works to match this distribution. It is scored on how far its own probabilities sit from the teacher's, and training pushes it to close that gap. In other words, the student learns the teacher's whole pattern of confidence rather than a single right answer, and that pattern is a stronger training signal than a one-word label. This is the core reason distillation works as well as it does. A single correct label discards the relationships between options, and soft labels keep them.

Read full article ↗excerpt · fair-use quotation