Audio · 2026-07-08 · 6 moments

Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

2 years after our first coverage, we return with Modal's other cofounder to explore why Agent Experience is working now, and everything they have learned building the new agent cloud. ✦ AI generated

Apple Podcasts ↗

GUEST Akshat Bubna HOST Alessio Fanelli HOST Shawn Wang (swyx)

timeline · colored by role

Context

Kubernetes was never designed for the bursty, resource-heavy, custom-image workloads that AI applications create — it's built for slow, steady web-server scaling — which is why Modal built a new runtime from scratch instead of using it.

Modal's CTO traces the company's founding insight: Kubernetes-based workflow orchestration was painful because Kubernetes is hard to manage and wasn't built for the burstiness and custom environments AI/ML workloads need.

transcript

Akshat Bubna: I first met Eric, who's the CEO, through an investor. Back then Eric was already thinking about building, a new runtime, and he got there thinking through why are workflow orchestration products so hard to use. It's because you have to run them on Kubernetes. Kubernetes is hard to manage. It's not built for burstiness and, custom images.

gives example · 1

Claim◆

The same principles that make developer experience good — self-provisioning infrastructure defined in code rather than YAML, with fast feedback loops — apply equally to agent experience, since an agent benefits even more than a human from not having to parse hundreds of untyped Kubernetes config files.

Modal has shifted its SDK team's focus from developer experience (DX) to agent experience (AX), arguing agents need the same self-provisioning, decorator-based infrastructure and fast feedback loops that made Modal appealing to human developers.

transcript

Akshat Bubna: We've changed our SDK team to think about agent experience instead of, developer experience and we think that the same benefits that apply for DX also apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and like write YAML that's not even typed when it can make a couple of changes in a decorator and it gets this self-provisioning runtime of, being able to see its changes live in action?

Claim

Even though fewer people read code directly anymore because agents write it, observability and good dashboards matter more than ever, since humans still need to interpret agent behavior and make judgment calls.

Akshat argues that as agents increasingly write and execute code that humans no longer read, strong observability becomes more important than ever so humans can still interpret what's happening and intervene.

transcript

Akshat Bubna: Like how good is your dashboard? And of course, like we have, we push a lot of it to the CLI so the agents can do their own investigation, but you still need humans to go interpret what's going on and, make judgment calls and whatnot. and that's I feel like, Maybe more important now than looking at the code itself.

Data◆

Reinforcement-learning rollout workloads are so extremely bursty that they can require spinning up on the order of a hundred thousand sandboxes at once, far more bursty than typical agent workloads.

Akshat notes that while ordinary agent sandbox usage isn't especially bursty, reinforcement-learning rollouts are an extreme exception, sometimes demanding roughly 100,000 sandboxes simultaneously.

transcript

Akshat Bubna: I agree that agents are not that bursty. sandboxes are, except when you're doing RL. RL is just insanely bursty. Yeah. Like when you're doing, rollouts, you sometimes need a hundred thousand sandboxes in your sandboxes.

Mechanism

Speculative decoding speeds up LLM inference by having a small draft model predict several tokens ahead, then having the large model verify them all in one batched pass — which is faster because batched verification is compute-bound rather than limited by memory bandwidth like standard one-token-at-a-time generation.

Akshat explains the mechanics behind speculative decoding: a smaller draft model proposes tokens ahead of time and the larger model verifies them in a batch, escaping the memory-bandwidth bottleneck of normal decoding.

transcript

Akshat Bubna: Speculative decoding is you have a smaller model, called a draft model, predict tokens ahead of the bigger model, and then you have the bigger model, verify all of this, all the tokens are predicted. And the reason it's faster is if you're predicting, one token at once, you're bound by memory bandwidth.

gives example · 1

Claim

Sandbox-level security must rely on hard, deterministic guardrails rather than trusting an LLM to mediate what an agent is allowed to do, because LLM-mediated permissions risk letting an agent exfiltrate data.

Pushing back on the idea of letting an LLM decide its own permissions, Akshat insists sandbox security needs hard, deterministic boundaries, with softer LLM-mediated rules layered on top only in addition.

transcript

Akshat Bubna: I'm, I'm skeptical of LLM media permission for stuff that is at the sandbox level because you do want hard boundaries. Otherwise, someone can exfiltrate stuff. I think you always need hard guardrails when you want, And you can pair those with softer guardrails, right?

gives example · 1

Highlight slides

From DX to AX✦ from: The same principles that make developer experience good — self-provisioning infrastructure defined in code rather than YAML, with fast feedback loops — apply equally to agent experience, since an agent benefits even more than a human from not having to parse hundreds of untyped Kubernetes config files.Why Agents Need It Even More✦ from: The same principles that make developer experience good — self-provisioning infrastructure defined in code rather than YAML, with fast feedback loops — apply equally to agent experience, since an agent benefits even more than a human from not having to parse hundreds of untyped Kubernetes config files.RL Rollouts Break the Bursty Scale✦ from: Reinforcement-learning rollout workloads are so extremely bursty that they can require spinning up on the order of a hundred thousand sandboxes at once, far more bursty than typical agent workloads.Why RL Is Different✦ from: Reinforcement-learning rollout workloads are so extremely bursty that they can require spinning up on the order of a hundred thousand sandboxes at once, far more bursty than typical agent workloads.

Related episodes

Video · a16z Podcast

Software in the Age of Agents | The a16z Show

→

Article · Simon Willison's Weblog

How I tricked Claude into leaking your deepest, darkest secrets

→

Audio · All-In Podcast

OpenAI CFO Sarah Friar: IPO, AI Rivalries, New Device, and Spending $100B+ on Compute

→