Article · 2026-07-04 · 5 moments

Better Models: Worse Tools

Better Models: Worse Tools Armin reports on a weird problem he ran into while hacking on Pi: The short version is that newer Claude models sometimes call Pi’s edit tool with extra, invented fields in the nested edits[] array. And not Haiku or some small model: Opus 4.8. The edit itself is usually correct but the arguments do not match the schema as the model invents made-up keys and Pi thus rejects the tool call and asks to try again. That alone is not too surprising as models emit malformed ✦ AI generated

Read original ↗

GUEST Armin Ronacher HOST Simon Willison

Context

Claude's edit tool is built around search-and-replace, whereas OpenAI's Codex uses an apply_patch mechanism instead, and OpenAI has publicly discussed training its models specifically to use that apply_patch tool well.

The author contrasts Anthropic's search-and-replace edit tool with OpenAI's apply_patch approach for Codex, noting OpenAI has spoken about deliberately training models to use its own patch mechanism.

transcript

Simon Willison: Claude's edit tool uses search and replace. OpenAI's Codex uses an apply_patch mechanism instead, and OpenAI have talked in the past about how their models are trained to use that tool effectively.

Claim◆

It is specifically the newest, state-of-the-art Claude models — Opus 4.8 and Sonnet 5 — that show this malformed-tool-call problem, while none of the older models do, meaning the SOTA models are worse at this particular tool schema than their older siblings.

Armin was surprised that the schema mistake gets worse, not better, with newer and more capable Claude models, since malformed tool calls are normally expected from weaker models rather than the SOTA ones.

transcript

Armin Ronacher: What surprised me is that this is getting worse with newer Anthropic models as both Opus 4.8 and Sonnet 5 show it but none of the older models. In other words, the SOTA models of the family are worse at this specific tool schema than their older siblings.

explains mechanism · 1

Mechanism◆

Newer Anthropic models have been specifically trained, presumably via reinforcement learning, to better use the edit tools baked into Claude Code, and this makes them more likely to misuse the differently-shaped custom edit tools of other coding harnesses like Pi.

Armin's theory for the regression: Anthropic has likely RL-trained recent models heavily on Claude Code's own edit tool, which inadvertently degrades their ability to correctly use differently-designed edit tools in third-party harnesses like Pi.

transcript

Armin Ronacher: Armin theorizes that this is because more recent Anthropic models have been specifically trained (presumably via Reinforcement Learning) to better use the edit tools that are baked into Claude Code. This has the unfortunate effect that other coding harnesses, such as Pi, may find that their own custom edit tools are more likely to be used incorrectly.

extends · 2gives example · 1supports · 2

Fact

Newer Claude models — including the flagship Opus 4.8, not just small models — sometimes call Pi's edit tool with extra, invented fields in the nested edits[] array, so the arguments fail to match the schema and Pi rejects the tool call and asks for a retry.

Armin Ronacher found that newer Claude models, even the flagship Opus 4.8, sometimes invent extra fields when calling Pi's edit tool, causing schema validation to fail and the call to be rejected.

transcript

Armin Ronacher: The short version is that newer Claude models sometimes call Pi's edit tool with extra, invented fields in the nested edits[] array. And not Haiku or some small model: Opus 4.8. The edit itself is usually correct but the arguments do not match the schema as the model invents made-up keys and Pi thus rejects the tool call and asks to try again.

explains mechanism · 1extends · 2

Prediction

Third-party coding harnesses like Pi may need to implement multiple edit tools so they can use whichever one performs best for the specific underlying model the user has selected.

The piece closes by floating the idea that tools like Pi might need to hedge against model-specific quirks by offering several edit-tool implementations, picking whichever suits the chosen model best.

transcript

Simon Willison: Does this mean third-party coding harnesses like Pi should implement multiple edit tools just so they can use the one with the best performance for the underlying model the user has selected?

provides context · 1

Highlight slides

SOTA Claude Models Fumble Tool Schema✦ from: It is specifically the newest, state-of-the-art Claude models — Opus 4.8 and Sonnet 5 — that show this malformed-tool-call problem, while none of the older models do, meaning the SOTA models are worse at this particular tool schema than their older siblings.Armin's Theory: Why Claude Regresses in Other Harnesses✦ from: Newer Anthropic models have been specifically trained, presumably via reinforcement learning, to better use the edit tools baked into Claude Code, and this makes them more likely to misuse the differently-shaped custom edit tools of other coding harnesses like Pi.Why This Surprised Armin✦ from: It is specifically the newest, state-of-the-art Claude models — Opus 4.8 and Sonnet 5 — that show this malformed-tool-call problem, while none of the older models do, meaning the SOTA models are worse at this particular tool schema than their older siblings.Side Effect: Misuse of Third-Party Edit Tools✦ from: Newer Anthropic models have been specifically trained, presumably via reinforcement learning, to better use the edit tools baked into Claude Code, and this makes them more likely to misuse the differently-shaped custom edit tools of other coding harnesses like Pi.

Related episodes

Article · Latent Space

[AINews] OpenAI reports median internal Codex output tokens grew 56x in Research, 32x in Customer Support, 27x in Engineering, and 13x in Legal since November 2025.

→

Article · Ahead of AI

Using Local Coding Agents

→

Audio · Dwarkesh Podcast

Eric Jang – Building AlphaGo from scratch

→