Maintaining a stable, append-only prompt prefix across iterations preserves prompt caching benefits. Any variation — such as tools serializing in different order from a hash map — silently breaks the cache and increases cost without any visible failure.

Prompt caching requires exact token-by-token matches at the start of the prompt. OpenAI discovered that Codex's use of an unordered hash map for MCP tool definitions caused tools to serialize in different orders, silently breaking the cache on every call. ✦ AI generated

Artificial Analysis · ByteByteGo Newsletter · 2026-07-29 · original ↗

LLM providers avoid repeated calculations with a technique called prompt caching. When a prompt arrives, the model reuses the cached internal state for the beginning of the prompt that matches a previous request (the prefix), and only computes the rest. The match is exact, token by token. If you change one token near the front of the prompt, everything after it must be recomputed. Appending to the end of the prompt keeps the cache valid. For the harness, this means the prompt it builds on every call must start with exactly the same bytes as the one before. That sounds trivial, but the harness rebuilds the request each time from its live in-memory state, so any small difference in how it assembles the prompt may silently break the match. OpenAI shared an example of this. Codex kept MCP tool definitions in a hash map, which does not guarantee ordering, so the same tools could serialize in a different order on each request. It was the same tools in the context, just in a different order. Codex was still completing tasks, just more expensively.

Read full article ↗excerpt · fair-use quotation