Pipeline parallelism during inference doesn't reduce compute time or memory-fetch time at all — it just relocates the memory fetch from one chip to another — so its only real benefit is reducing per-GPU memory capacity requirements, not latency or throughput.

Responding to Dwarkesh's mention of Ilya Sutskever's remark that 'pipelining is not wise,' Reiner Pope clarifies that pipeline parallelism in inference saves memory capacity only, not runtime, and that this benefit further cancels out for the KV cache because sharding across pipeline stages requires proportionally more sequences in flight. ✦ AI generated

Reiner Pope · Dwarkesh Podcast · 2026-04-29 · original ↗

In inference, what are we saving on? Are we saving on memory time or compute time? Not really. We're just moving the memory time from one chip to another chip, or one rack to a different rack. There's no actual benefit in runtime. However, what we are saving on is memory capacity.

verbatim transcript · starts at 63:37