Definition◆Article
An LLM is a neural network trained to predict the next token. It cannot run a shell command, edit a file, or remember anything between calls. Agent tasks like fixing bugs require a three-layer system — harness, API, and inference — to bridge token prediction and real-world actions.
The article explains why AI applications like Codex are not just LLMs but systems with three layers (harness, API, inference), because an LLM on its own can only predict tokens and cannot execute actions or maintain state between calls. ✦ AI generated
Artificial Analysis · ByteByteGo Newsletter · 2026-07-29 · original ↗
An LLM is a neural network trained to predict the next token. It takes a sequence of tokens as input and produces a sequence of tokens as output. It cannot run a shell command, edit a file, or remember anything between calls. But an agent task like "fix this bug and run the tests" is mostly actions. Something has to turn the model's predicted tokens into real commands, feed the results back, and continue until the task is done. That is the job of the harness layer, a system built on top of the LLM to handle those responsibilities. It takes the user's task as input, decides which instructions, which tool definitions, and how much history to include in the context, and maintains the conversation history. When the model responds with a tool call, the harness executes it under approval policies in a sandbox environment, appends the result, and sends the conversation back to the LLM.
Read full article ↗excerpt · fair-use quotation
- ·A neural network trained to predict the next token
- ·Takes tokens in, produces tokens out
- ·Cannot run shell commands, edit files, or remember state between calls
- ·Fixing a bug means mostly actions, not token prediction
- ·Something must bridge token outputs and real commands
- ·Three-layer system required: harness, API, inference
- ·Built on top of the LLM to manage real-world actions
- ·Selects instructions, tool definitions, and history for context
- ·Executes tool calls in a sandbox and feeds results back to the model
Around this claim
This moment responds to
provides context → Using persistent WebSockets instead of HTTPS and sending only incremental changes instead of full payloads eliminates the repeated TCP/TLS handshake costs and growing payload bloat that plague multi-call agent loops.Artificial Analysis · ByteByteGo Newsletterexplains mechanism → Multi-agentic architecture is needed when a large, complex goal must be broken down into discrete steps, with each step handled by a specific agent tasked with executing that narrow task before handing off to the next.Rashmi Shetty · The TWIML AI Podcastprovides context → Maintaining a stable, append-only prompt prefix across iterations preserves prompt caching benefits. Any variation — such as tools serializing in different order from a hash map — silently breaks the cache and increases cost without any visible failure.Artificial Analysis · ByteByteGo Newsletterprovides context → Instead of including all available tool schemas in every prompt, OpenAI defers tool discovery — loading schemas on demand via BM25 search — and in Code Mode lets the model write a JavaScript program that executes multiple tool calls in parallel, reducing round trips and keeping intermediate data out of context.Artificial Analysis · ByteByteGo Newsletterprovides context → The API layer avoids full retokenization on every loop iteration by storing tokenized state server-side over WebSockets, and hides safety check latency by running classifiers in parallel with inference, using the prompt processing window that would otherwise be idle.Artificial Analysis · ByteByteGo Newsletterprovides context → The biggest inference gains come from cache-aware routing, production-informed KV cache management, speculative decoding with a draft model, and separating prefill from decode. The overarching lesson is that no single optimization is a game changer on its own — big wins come from chaining many small ones together, and from testing with real production traffic patterns.Artificial Analysis · ByteByteGo Newsletter