Cloudflare's pay-per-crawl shifted from counting crawls to measuring use, because a crawl is a weak measure of value: a page might be crawled once and cited in thousands of answers or crawled repeatedly and cited in none.

Pay Per Crawl lets a website allow a crawler, block it, or charge it a flat per-request price, with Cloudflare handling the billing as the merchant of record. ... A year on, Cloudflare made an adjustment to its own model. The company argued that a crawl is a weak measure of value, because a single page might be crawled once and then cited in thousands of AI answers, or crawled repeatedly and cited in none. It backed the argument with a figure from its own network, that more than half of the crawl traffic from well-behaved bots goes to re-fetching pages that have stayed the same since the last visit. Counting crawls, then, counts something that only loosely tracks the value delivered. So the unit of payment began to move from the crawl toward the use, an approach Cloudflare describes as Pay Per Use. It is candidly framed as an experiment at this point.

Read full article ↗excerpt · fair-use quotation