Inkling separates the cost of storing a model from the cost of running it by activating only 6 of 256 experts per token, processing roughly 41 billion of its 975 billion parameters at a time.

Inkling separates the cost of storing a model from the cost of running it. This is the reason a model this large is affordable to use. In an ordinary transformer, the feed-forward step in each layer is a single network, and every token passes through all of it. If that network holds 5 billion parameters, then every token processed involves all 5 billion. Inkling replaces that single network with 256 smaller ones, called experts. For each token, a selection step picks six of the 256. Only those six run, and the remaining 250 sit idle for that token while handling other tokens instead. This design pattern is called Mixture of Experts, and Thinking Machines states that their version largely follows the approach published by DeepSeek. The total across the whole model is 975 billion parameters, and the count involved in processing any single token is roughly 41 billion. That is about 4 percent of the model running at a time.

Read full article ↗excerpt · fair-use quotation