MechanismArticle

Gemma 4's per-layer embeddings give each transformer layer its own small token-specific vector, adding representational capacity without scaling the whole transformer stack to a larger parameter count.

PLE is a separate efficiency trick from KV-sharing: it stores extra per-layer, per-token embedding capacity that is cheap to look up, letting Gemma 4's 'E' models have more effective capacity than their 'active' parameter count suggests. ✦ AI generated

Sebastian Raschka · Ahead of AI · 2026-05-16 · original ↗

Share card ↓Export .md Export .txt

The PLE design keeps the expensive transformer blocks closer to the smaller "effective" size, while storing additional capacity in per-layer embedding tables. These are much cheaper to use than adding more attention or FFN weights, since they are mainly lookup-style parameters that can be cached.

Read full article ↗excerpt · fair-use quotation

▦ Google ✳ Gemma 4

Related moments

Example◆Video · 15:12 · 2m

When Osmo's learned odor embedding space was visualized in two dimensions, molecules with the same human-labeled smell clustered together, and finer distinctions like jasmine, rose, and violet emerged as nested sub-regions inside the broader floral cluster without ever being told that hierarchy existed.

Alex Wiltschko · The TWIML AI Podcast

Example◆Video · 14:51 · 2m

The learned 'principal odor map' embedding spontaneously organizes molecules into nested perceptual neighborhoods — for example, jasmine, rose, and violet form sub-regions inside a larger floral region — without ever being told this hierarchy exists.

Alex Wiltschko · The TWIML AI Podcast

Example◆Video · 22:54 · 2m

Word embedding vectors trained via word2vec capture real semantic relationships, like gender and royalty, purely from word co-occurrence statistics, without anyone explicitly teaching the model those concepts.

CJ · Syntax