Thinking Machines prevents routing collapse by adding a per-expert bias that shifts selection decisions but never affects output weighting, keeping expert usage balanced without injecting a competing gradient into training.

If we run that loop long enough, we can end up with a 256-expert layer where perhaps 20 experts handle nearly everything and the other 236 stay underdeveloped. This is called routing collapse. In other words, while we paid to store 256 experts, we got the capability of just 20. There is a second cost too. Since experts are usually spread across different machines, a machine holding four popular experts can become a bottleneck while its neighbours idle. The traditional fix for this adds a penalty to the training objective that grows when expert usage is uneven. This means training the model on two things at once: predicting the next token correctly, and keeping expert usage balanced. The trouble is that these two goals produce gradients pointing in different directions. Thinking Machines uses a method introduced by Wang and colleagues, which was also adopted by DeepSeek. This method removes the conflict entirely by keeping a separate bias value for each expert, which is just a small number added to that expert's score. The trick is where the bias gets applied. Expert 47 has been overloaded recently, so its bias has drifted down to −0.15. This drops its selection score below expert 88's, and expert 88 takes the slot instead. The bias changed which expert got picked. When expert 88's output is combined into the final result, it is weighted by 0.80, its original router score, with the bias left out. In other words, the bias affects selection only. It never touches the weighting, and it is updated by a simple counting rule that runs outside backpropagation entirely.

Read full article ↗excerpt · fair-use quotation