Audio · 2026-06-24 · 6 moments

Why the Frontier Ecosystem must be Open — Matei Zaharia and Reynold Xin, Databricks

In a rare double-interview, the Databricks technical leaders riff on what it will take for every company to build Agent Clouds ✦ AI generated

Apple Podcasts ↗

timeline · colored by role

Claim◆

Coding agents and custom enterprise agents run into the exact same infrastructure problems — model/harness portability, session sharing, and security — so they should be built on one common layer instead of being treated as separate categories.

Matei explains that Omnigent emerged from noticing internal coding-agent tooling and custom enterprise agents kept hitting identical problems — switching models and harnesses, sharing sessions, security — so Databricks built one common layer to serve both.

transcript

Matei Zaharia: at first people thought it was weird. They're like, "Why are you doing coding agents and custom agents in the same thing?" But I said it's, it's the same problems and, you just wanna build the stuff that lets you deliver the agent, maybe control it if you care about security, and, make it portable across things.

Mechanism

A layer should be open-sourced when it stands to benefit from network effects and third-party integrations — a competing open alternative will win out over a closed one in the long run.

Matei argues Databricks open-sources layers like Spark and Omnigent specifically because they benefit from network effects — outside contributions and integrations — and that an open competitor would eventually beat a closed equivalent.

transcript

Matei Zaharia: imagine, we our thing wasn't open. We had some agent hosting thing, but it's not open and then there is an open one. Which one's gonna win in the long run? So like here, because there is this benefit from like people writing integrations, it'll be, it'll be that.

supports · 1gives example · 1

Example

Simple yes/no allow-lists for agent tool permissions can't capture real risk; agent security needs stateful, contextual policies that track session history so a combination of individually-fine actions can be blocked when risky together.

Matei explains that binary tool-permission rules fail because actions that are fine alone (reading a doc, installing a package) become dangerous in combination via prompt injection, so Omnigent tracks session state to make contextual security decisions.

transcript

Matei Zaharia: should my agent be able to read, some confidential documents, or let's say, should it be able to install new packages from npm, which, maybe it's compromised. Yes or no? Like, maybe I wanna allow it. Should my agent be able to publish stuff to the company website? Well, if I'm using it to code on the website, yes. But should it be able to do both, so it can, like grab a confidential document and be prompt injected and leak it? Probably not.

Anecdote

Because agent sessions can track cumulative dollar spend, users can cap a session's spending and require explicit approval before it spends more, preventing runaway costs.

Matei describes capping per-session agent spend (e.g., $5) with an approval prompt before going over, after one of his own debugging sessions burned $500 in tokens reading log files.

transcript

Matei Zaharia: I've had, like, I ask an agent to debug something, and it spent $500 because it decided to read a lot of log files and burn a lot of tokens. but I can literally say, "Okay, launch a agent to do this and cap it to spending $5." Like, ask me for permission if it needs more.

Claim

Change data capture (CDC) pipelines, despite being an industry standard, are so brittle that they deserve the joking rename 'continuous data corruption' because schema changes on the source database routinely break them.

Reynold jokes that CDC — used industry-wide to replicate transactional data into analytics systems — is so fragile that it should be called "continuous data corruption," since an OLTP schema change routinely breaks the pipeline.

transcript

Reynold Xin: But it's so brittle that, we joke that it's, should be called continuous data corruption, because you might change your schema on your OLTP database, and then the CDC pipeline fails to handle the schema change.

Mechanism◆

Rather than building one unified engine for both transactional and analytical workloads (the historically failed HTAP dream), unifying only the storage layer — writing transactional data in column-oriented format — captures nearly all the benefit without the compromises of a single engine.

Reynold lays out Databricks' LTAP thesis: instead of chasing HTAP's failed goal of one engine for both OLTP and OLAP, unifying just the storage layer so Postgres data lands directly in column format gets 99% of the benefit with no replication lag.

transcript

Reynold Xin: Our whole idea of LTAP, it's obviously a wordplay on the term HTAP, is that we think this is HTAP done right. HTAP wants to build a single engine for both. We think you can get 99% of what you need by unifying the storage, and just have a single storage layer. And once you have the single storage layer, if your Postgres databases are writing data in a column-oriented format, everything analytics can just go read that data directly without any delay.

provides context · 1

Highlight slides

One Layer for All Agents✦ from: Coding agents and custom enterprise agents run into the exact same infrastructure problems — model/harness portability, session sharing, and security — so they should be built on one common layer instead of being treated as separate categories.Why Treat Them as One Category✦ from: Coding agents and custom enterprise agents run into the exact same infrastructure problems — model/harness portability, session sharing, and security — so they should be built on one common layer instead of being treated as separate categories.LTAP: HTAP Done Right✦ from: Rather than building one unified engine for both transactional and analytical workloads (the historically failed HTAP dream), unifying only the storage layer — writing transactional data in column-oriented format — captures nearly all the benefit without the compromises of a single engine.Column Storage, No Delay✦ from: Rather than building one unified engine for both transactional and analytical workloads (the historically failed HTAP dream), unifying only the storage layer — writing transactional data in column-oriented format — captures nearly all the benefit without the compromises of a single engine.

Related episodes

Article · Simon Willison's Weblog

Coding agents and custom enterprise agents run into the exact same infrastructure problems — model/harness portability, session sharing, and security — so they should be built on one common layer instead of being treated as separate categories.

A layer should be open-sourced when it stands to benefit from network effects and third-party integrations — a competing open alternative will win out over a closed one in the long run.

Simple yes/no allow-lists for agent tool permissions can't capture real risk; agent security needs stateful, contextual policies that track session history so a combination of individually-fine actions can be blocked when risky together.

Because agent sessions can track cumulative dollar spend, users can cap a session's spending and require explicit approval before it spends more, preventing runaway costs.

Change data capture (CDC) pipelines, despite being an industry standard, are so brittle that they deserve the joking rename 'continuous data corruption' because schema changes on the source database routinely break them.

Rather than building one unified engine for both transactional and analytical workloads (the historically failed HTAP dream), unifying only the storage layer — writing transactional data in column-oriented format — captures nearly all the benefit without the compromises of a single engine.

Better Models: Worse Tools

sqlite-utils 4.0rc2, mostly written by Claude Fable (for about $149.25)

Using Local Coding Agents