Article · 2026-08-11 · 6 moments

How Cloudflare Is Making AI Pay for Content

In this article, we will go through Cloudflare’s solution in the following five steps. ✦ AI generated

Mechanism

Because Cloudflare operates as a reverse proxy for a large portion of the web, a request is read, classified, checked, and acted on at the edge before it reaches the origin, and this position is more general-purpose than caching.

Cloudflare's reverse proxy position lets it read and act on requests before they reach the origin server, a capability that is more general-purpose than mere caching.

transcript

Author: A reverse proxy is a server that stands in front of other servers and receives requests on their behalf. Since Cloudflare operates as a reverse proxy for a large portion of the web, a request headed for one of those sites reaches Cloudflare's network first and passes through it on the way to the origin. ... the position of a proxy is more general-purpose than caching. Caching stores a copy of a response so it can be served quickly the next time, and it is just one of the many useful things a proxy can do. From the same middle position, a request can also be read, classified, checked, and acted on before it continues to the origin.

Context

Charging an anonymous caller a fraction of a cent for a single request was impractical because the collection cost exceeded the payment itself, since existing usage-based charging (cloud services, APIs) only worked for known buyers with API keys.

The web already charged by usage for known buyers, but collecting a fraction of a cent from an anonymous caller cost more than the payment was worth, making micro-payment to anonymous traffic impractical.

transcript

Author: parts of the Internet were already charging by usage before any of this. Cloud services and APIs have been sold by the call and by the hour for years, though only to a buyer the seller already knew, who signed up and received an API key. Charging an anonymous caller a fraction of a cent for a single request stayed impractical, because collecting such a small payment once cost more than the payment returned.

Mechanism◆

Cloudflare resolves identity, permission, and payment inside a single request at the edge using Web Bot Auth for cryptographic identity, behavior classification for permission, and the x402 protocol for payment, so the origin only receives requests after all three are settled.

The x402 protocol settles identity, permission, and payment inside a single HTTP request at the edge, running a short state-machine exchange where a 402 response states a price, the client re-sends with proof of payment, and a facilitator verifies it — removing the signup step per-seat licensing and API keys required.

transcript

Author: The design goal for recent changes is to resolve identity, permission, and payment inside a single request, at the edge, before the origin responds. ... Identity: The traditional identifier, the User-Agent string, can be set to any value by the caller, so it offers weak assurance. Cloudflare's answer is Web Bot Auth, an authentication method that uses cryptographic signatures in HTTP messages to verify that a request comes from a particular automated source. ... Payment: This comes last and is attached to the request through x402. ... The exchange runs through a short sequence that can be compared to a small state machine. A client requests a resource that sits behind a price. Rather than returning the resource, the server responds with 402 and a small payload stating the price, the accepted asset, and where to pay. The client re-sends the same request with proof of payment attached. A facilitator verifies the payment, and the server returns the resource. Two properties make this suitable for machine traffic. The payment amounts can be very small because the protocol adds almost no overhead to the request. And the payment itself serves as the credential, so a buyer with no prior relationship can access the content by showing the proof of payment.

Mechanism◆

Cloudflare's pay-per-crawl shifted from counting crawls to measuring use, because a crawl is a weak measure of value: a page might be crawled once and cited in thousands of answers or crawled repeatedly and cited in none.

Cloudflare initially offered a block / allow / charge-per-crawl control, then argued that a crawl is a weak measure of value and shifted toward pay-per-use, candidly framed as an experiment because usage outcomes are harder to measure.

transcript

Author: Pay Per Crawl lets a website allow a crawler, block it, or charge it a flat per-request price, with Cloudflare handling the billing as the merchant of record. ... A year on, Cloudflare made an adjustment to its own model. The company argued that a crawl is a weak measure of value, because a single page might be crawled once and then cited in thousands of AI answers, or crawled repeatedly and cited in none. It backed the argument with a figure from its own network, that more than half of the crawl traffic from well-behaved bots goes to re-fetching pages that have stayed the same since the last visit. Counting crawls, then, counts something that only loosely tracks the value delivered. So the unit of payment began to move from the crawl toward the use, an approach Cloudflare describes as Pay Per Use. It is candidly framed as an experiment at this point.

Mechanism

Cloudflare's taxonomy classifies automated traffic by behavior into three types — search (building an index that sends referral visitors back), agent (acting in real time on a person's behalf), and training (absorbing content into a model) — which appear identical in a raw request log yet carry very different business consequences.

Cloudflare sorts automated traffic by behavior rather than a single 'AI' label, separating search (referral-generating), agent (real-time assistant), and training (content absorption) despite them looking identical in raw logs.

transcript

Author: Cloudflare's taxonomy groups automated traffic by behavior rather than by the single label 'AI'. Three behaviors impact the key policy decisions. Search covers behavior that builds an index of a site so an engine can answer questions about it later. ... An agent covers behavior that acts in real time on a person's behalf, usually with a human waiting for the result. ... Training covers behavior that takes content to train or fine-tune a model, where the content is absorbed into the model rather than pointing a visitor back. These three appear identical in a raw request log, yet they carry very different consequences for a site's business.

Context

For most of the web's history, websites earned value after a request (through ads, subscriptions, or return visits), but agent traffic that acts on a person's behalf, skips ads, and completes its task in a single pass leaves those downstream settlement points without value to collect.

Agent traffic hollows out the web's historic model of settling value through human attention downstream of a request, since software visits once, skips ads, and completes its task in a single pass.

transcript

Author: For most of its history, websites made money after a request rather than during it. A browser asked for a page, and the server returned it at no charge. The value arrived later, once a person saw an advertisement, bought a subscription, or came back for another visit. ... An agent is software that acts on a person's behalf, which in practice means it requests a page or a data feed once, takes what it needs, and finishes in a single pass. It moves past advertisements, operates outside any subscription, and completes its task before a site has a chance to earn from that attention. ... This means that request volume climbs while revenue stays flat.

Highlight slides

Cloudflare's shift from pay-per-crawl to pay-per-use✦ from: Cloudflare's pay-per-crawl shifted from counting crawls to measuring use, because a crawl is a weak measure of value: a page might be crawled once and cited in thousands of answers or crawled repeatedly and cited in none.Resolving identity, permission & payment in a single edge request✦ from: Cloudflare resolves identity, permission, and payment inside a single request at the edge using Web Bot Auth for cryptographic identity, behavior classification for permission, and the x402 protocol for payment, so the origin only receives requests after all three are settled.x402: payment as a request-time state machine✦ from: Cloudflare resolves identity, permission, and payment inside a single request at the edge using Web Bot Auth for cryptographic identity, behavior classification for permission, and the x402 protocol for payment, so the origin only receives requests after all three are settled.

Related episodes

Article · Simon Willison's Weblog

Don't classify. Hallucinate!

→

Video · Syntax

Mosh, Caddy & Tailscale: A Remote Dev Deep Dive

→

Audio · All-In Podcast

OpenAI CFO Sarah Friar: IPO, AI Rivalries, New Device, and Spending $100B+ on Compute

→