The API layer avoids full retokenization on every loop iteration by storing tokenized state server-side over WebSockets, and hides safety check latency by running classifiers in parallel with inference, using the prompt processing window that would otherwise be idle.

With WebSockets, the API tokenizes only the new items and appends them to the stored sequence, making per-call tokenization O(1). Safety checks run simultaneously with inference, hiding inside the time the model already spends processing the prompt. ✦ AI generated

Artificial Analysis · ByteByteGo Newsletter · 2026-07-29 · original ↗

The fix is to run the safety checks and inference at the same time. The model takes some time to process the prompt before its first token comes out, so the checks use that window to finish. If a check fails, the API reacts depending on the model. For some models, it streams tokens to the user right away and cuts the stream when a check fails. For more sensitive models, it holds the output until the checks pass and only then releases it. In both cases, the time spent on safety hides inside a wait that was going to happen anyway.

Read full article ↗excerpt · fair-use quotation