Inkling uses relative position encoding instead of the standard RoPE, encoding the distance between token pairs rather than absolute positions so that pairs at any distance use learned values the model has already seen during training.

Thinking Machines chose a relative position encoding scheme over the now-standard RoPE, because it lets the model handle token distances far beyond what appeared in training without extrapolation problems. ✦ AI generated

Thinking Machines · ByteByteGo Newsletter · 2026-08-18 · original ↗

Thinking Machines picked an older technique over the current standard. This is due to the lengths the model never saw during training. Almost every recent open model uses Rotary Position Embedding, shortened to RoPE. Each token's query and key are treated as points that get rotated by an angle proportional to that token's position in the sequence. However, those rotation angles were only ever encountered at positions the model actually trained on. If training used sequences up to 32,000 tokens, then every angle the model learned to interpret came from that range. If we ask it about position 900,000, the angle involved falls outside anything it has experience with. Inkling uses a relative scheme in the style of Shaw and colleagues. Rather than encoding where each token sits, this approach learns a value for each distance between two tokens and adds that value directly to the comparison score. For example, tokens at positions 5 and 9 are 4 apart. Tokens at positions 500,005 and 500,009 are also 4 apart. A relative scheme treats both pairs identically, because 4 is 4 wherever it occurs. Distances beyond some cutoff, say anything more than 128 apart, all share the same learned value, so a pair 900,000 tokens apart uses a value the model has seen countless times during training. Nothing has to be extrapolated. Thinking Machines states that this performed better and extrapolated better to longer sequences than RoPE in their testing.

Read full article ↗excerpt · fair-use quotation