Article · 2026-07-29 · 6 moments

How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference

To understand what techniques are adopted in frontier labs to make AI applications more efficient, we met with the OpenAI engineers who developed and shipped various efficiency techniques into the systems behind Codex and ChatGPT Work. ✦ AI generated

Read original ↗

GUEST Artificial Analysis

Definition◆

An LLM is a neural network trained to predict the next token. It cannot run a shell command, edit a file, or remember anything between calls. Agent tasks like fixing bugs require a three-layer system — harness, API, and inference — to bridge token prediction and real-world actions.

The article explains why AI applications like Codex are not just LLMs but systems with three layers (harness, API, inference), because an LLM on its own can only predict tokens and cannot execute actions or maintain state between calls.

transcript

Artificial Analysis: An LLM is a neural network trained to predict the next token. It takes a sequence of tokens as input and produces a sequence of tokens as output. It cannot run a shell command, edit a file, or remember anything between calls. But an agent task like "fix this bug and run the tests" is mostly actions. Something has to turn the model's predicted tokens into real commands, feed the results back, and continue until the task is done. That is the job of the harness layer, a system built on top of the LLM to handle those responsibilities. It takes the user's task as input, decides which instructions, which tool definitions, and how much history to include in the context, and maintains the conversation history. When the model responds with a tool call, the harness executes it under approval policies in a sandbox environment, appends the result, and sends the conversation back to the LLM.

Mechanism

Instead of including all available tool schemas in every prompt, OpenAI defers tool discovery — loading schemas on demand via BM25 search — and in Code Mode lets the model write a JavaScript program that executes multiple tool calls in parallel, reducing round trips and keeping intermediate data out of context.

OpenAI keeps hundreds of tool definitions out of the prompt by having the model search for tools when needed (deferred discovery), and in Code Mode the model writes JavaScript programs that can fan out independent tool calls in parallel, returning only the compact final result.

transcript

Artificial Analysis: With Code Mode, instead of emitting tool calls one by one, the model writes a small program that makes those calls. The harness runs this program in an embedded JavaScript runtime, where every tool is available as a function. The script can fan out independent calls in parallel, filter and join the results in plain code, and return only the compact answer. This way, the intermediate data stays in the runtime, and only the final result enters the context.

provides context · 1

Mechanism

Maintaining a stable, append-only prompt prefix across iterations preserves prompt caching benefits. Any variation — such as tools serializing in different order from a hash map — silently breaks the cache and increases cost without any visible failure.

Prompt caching requires exact token-by-token matches at the start of the prompt. OpenAI discovered that Codex's use of an unordered hash map for MCP tool definitions caused tools to serialize in different orders, silently breaking the cache on every call.

transcript

Artificial Analysis: LLM providers avoid repeated calculations with a technique called prompt caching. When a prompt arrives, the model reuses the cached internal state for the beginning of the prompt that matches a previous request (the prefix), and only computes the rest. The match is exact, token by token. If you change one token near the front of the prompt, everything after it must be recomputed. Appending to the end of the prompt keeps the cache valid. For the harness, this means the prompt it builds on every call must start with exactly the same bytes as the one before. That sounds trivial, but the harness rebuilds the request each time from its live in-memory state, so any small difference in how it assembles the prompt may silently break the match. OpenAI shared an example of this. Codex kept MCP tool definitions in a hash map, which does not guarantee ordering, so the same tools could serialize in a different order on each request. It was the same tools in the context, just in a different order. Codex was still completing tasks, just more expensively.

explains mechanism · 1provides context · 1

Mechanism

The API layer avoids full retokenization on every loop iteration by storing tokenized state server-side over WebSockets, and hides safety check latency by running classifiers in parallel with inference, using the prompt processing window that would otherwise be idle.

With WebSockets, the API tokenizes only the new items and appends them to the stored sequence, making per-call tokenization O(1). Safety checks run simultaneously with inference, hiding inside the time the model already spends processing the prompt.

transcript

Artificial Analysis: The fix is to run the safety checks and inference at the same time. The model takes some time to process the prompt before its first token comes out, so the checks use that window to finish. If a check fails, the API reacts depending on the model. For some models, it streams tokens to the user right away and cuts the stream when a check fails. For more sensitive models, it holds the output until the checks pass and only then releases it. In both cases, the time spent on safety hides inside a wait that was going to happen anyway.

explains mechanism · 1provides context · 1

Claim◆

The biggest inference gains come from cache-aware routing, production-informed KV cache management, speculative decoding with a draft model, and separating prefill from decode. The overarching lesson is that no single optimization is a game changer on its own — big wins come from chaining many small ones together, and from testing with real production traffic patterns.

OpenAI optimizes inference through cache-aware routing, KV cache management based on real usage data, speculative decoding where a small draft model proposes tokens for the large model to verify, and separating prefill and decode onto different hardware. The team emphasizes that no single technique is transformative on its own.

transcript

Artificial Analysis: The inference team told us that every time they picked one favorite technique to focus on, they regretted it. Focusing on one part of the stack made them underinvest in the others. No single optimization is a game changer on its own. The big wins come from chaining many small ones together. They also learned to test with the same traffic shapes that production actually serves, because a change that looks like a win offline can hurt in real traffic.

provides context · 2

Mechanism

Using persistent WebSockets instead of HTTPS and sending only incremental changes instead of full payloads eliminates the repeated TCP/TLS handshake costs and growing payload bloat that plague multi-call agent loops.

Codex opens a single WebSocket to the API and keeps it alive across all model calls in a turn, eliminating repeated connection setup. It also sends only new items with a reference to the previous response, so payload size stays small.

transcript

Artificial Analysis: The fix for the connection cost is to open one connection and keep it alive, instead of creating a new one for every call. This is what WebSockets are designed for. A WebSocket needs just one initial handshake, and after that, both sides can send messages whenever they want with no per-message setup. Codex opens a single WebSocket to the API and keeps it open across all the model calls in a turn. This eliminates the repeated TCP and TLS setup. The fix for the payload repetition is to stop resending what the server already knows. The harness keeps the previous request and the completed response. If nothing but the conversation input has changed, it sends only the new items along with a reference to the previous response.

provides context · 1

Highlight slides

What an LLM Actually Is✦ from: An LLM is a neural network trained to predict the next token. It cannot run a shell command, edit a file, or remember anything between calls. Agent tasks like fixing bugs require a three-layer system — harness, API, and inference — to bridge token prediction and real-world actions.No Single Optimization Is a Game Changer✦ from: The biggest inference gains come from cache-aware routing, production-informed KV cache management, speculative decoding with a draft model, and separating prefill from decode. The overarching lesson is that no single optimization is a game changer on its own — big wins come from chaining many small ones together, and from testing with real production traffic patterns.Agent Tasks Need More Than an LLM✦ from: An LLM is a neural network trained to predict the next token. It cannot run a shell command, edit a file, or remember anything between calls. Agent tasks like fixing bugs require a three-layer system — harness, API, and inference — to bridge token prediction and real-world actions.Four Key Techniques for Inference Gains✦ from: The biggest inference gains come from cache-aware routing, production-informed KV cache management, speculative decoding with a draft model, and separating prefill from decode. The overarching lesson is that no single optimization is a game changer on its own — big wins come from chaining many small ones together, and from testing with real production traffic patterns.The Harness Layer✦ from: An LLM is a neural network trained to predict the next token. It cannot run a shell command, edit a file, or remember anything between calls. Agent tasks like fixing bugs require a three-layer system — harness, API, and inference — to bridge token prediction and real-world actions.Test With Real Production Traffic✦ from: The biggest inference gains come from cache-aware routing, production-informed KV cache management, speculative decoding with a draft model, and separating prefill from decode. The overarching lesson is that no single optimization is a game changer on its own — big wins come from chaining many small ones together, and from testing with real production traffic patterns.

Related episodes

Video · The Cognitive Revolution

Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

→

Video · Syntax

I Built an LLM From Scratch

→

Audio · No Priors

AI for Atoms: How Periodic Labs is Revolutionizing Materials Engineering with Co-Founder Liam Fedus

→