Inkling processes images and audio inline without separately pretrained encoders, converting audio via dMel rounding and images via 40x40 pixel patches through a lightweight hMLP stem, with all components trained from scratch alongside the language model.

Inkling handles images and audio without dedicated pretrained encoders, using simple inline conversions (dMel for audio, patch stems for images) that are trained together with the rest of the model from scratch. ✦ AI generated

Thinking Machines · ByteByteGo Newsletter · 2026-08-18 · original ↗

Everything so far concerns text moving through the model. Images and audio have to get in first, and Inkling accepts both without a separately trained encoder standing in front of it. Sound arrives as a mel spectrogram, which is a standard way of representing audio as a grid of numbers. The dMel method then rounds each of those loudness values to one of a fixed set of levels, in the same way you might round 0.73 to 0.7. That is the entire conversion. No separate audio model needs training beforehand, because rounding numbers requires no training. In the case of images, an image is cut into square patches measuring 40 by 40 pixels. A 400 by 400 pixel image therefore becomes 100 patches. Each patch passes through a small four-stage network called an hMLP stem, which combines the pixels within that patch and processes each patch independently of every other one. Both then pass through a lightweight conversion layer and join the text tokens in a single sequence, processed by the same 66 layers we have been talking about. Thinking Machines states that these multimodal components were trained from scratch on general-domain data, meaning they learned alongside the rest of the model rather than arriving pretrained.

Read full article ↗excerpt · fair-use quotation