Gemma 4's per-layer embeddings give each transformer layer its own small token-specific vector, adding representational capacity without scaling the whole transformer stack to a larger parameter count.

PLE is a separate efficiency trick from KV-sharing: it stores extra per-layer, per-token embedding capacity that is cheap to look up, letting Gemma 4's 'E' models have more effective capacity than their 'active' parameter count suggests. ✦ AI generated

Sebastian Raschka · Ahead of AI · 2026-05-16 · original ↗

The PLE design keeps the expensive transformer blocks closer to the smaller "effective" size, while storing additional capacity in per-layer embedding tables. These are much cheaper to use than adding more attention or FFN weights, since they are mainly lookup-style parameters that can be cached.

Read full article ↗excerpt · fair-use quotation