MechanismArticle

Inkling uses relative positional encoding / relative attention bias instead of RoPE, which multiple technical observers called one of the most novel large-scale architecture choices in the release.

Community technical analysts highlight that Inkling forgoes the now-standard RoPE positional encoding in favor of relative attention bias, seen as a bold, closely-watched architecture bet at this scale. ✦ AI generated

eliebakouch · Latent Space · 2026-07-16 · original ↗

Share card ↓Export .md Export .txt

Relative positional encoding / relative attention bias instead of RoPE; multiple posters called this one of the most novel large-scale choices.

Read full article ↗excerpt · fair-use quotation

✳ Inkling ✳ RoPE

Around this claim

Evidence · 1

Inkling uses relative position encoding instead of the standard RoPE, encoding the distance between token pairs rather than absolute positions so that pairs at any distance use learned values the model has already seen during training.Thinking Machines · ByteByteGo Newsletter · conf 92%

This moment responds to

extends → Inkling is Thinking Machines Lab's first general model: fully open-weight, 975B total parameters, natively multimodal, and available on Tinker, Hugging Face, and partner platforms.Soumith Chintala · Latent Space