Inkling alternates 55 sliding-window layers and 11 full-attention layers in a 5:1 ratio, allowing it to support a one-million-token context window where long-range information travels through the sparse full-attention layers while most layers process only local context cheaply.

Inkling's attention mechanism uses mostly local sliding-window layers with a smaller set of full-attention layers interleaved, making a million-token context window computationally feasible while still allowing long-range information flow. ✦ AI generated

Thinking Machines · ByteByteGo Newsletter · 2026-08-18 · original ↗

Inkling supports a context window of one million tokens, and a trillion comparisons per layer across 66 layers is far beyond what any reasonable amount of hardware can deliver. This is where a sliding-window layer restricts each token to a fixed number of recent tokens rather than everything before it. Inkling alternates between sliding-window layers and full-attention layers at a ratio of 5:1. Integration notes published by the vLLM project put concrete numbers on the split, describing the 66 layers as 55 sliding-window layers and 11 full-attention layers. Information from far away travels through those eleven full layers. For example, picture a fact at token 200 that matters for a prediction at token 900,000. In layer 6, the first full-attention layer, the representation at position 900,000 can reach back and pick up that fact directly. From there, it rides forward within the local representations, is refreshed at layer 12, again at layer 18, and so on. The long-range path exists on roughly one layer in six, and the other five handle nearby context at a fraction of the price.

Read full article ↗excerpt · fair-use quotation