What is pay per crawl?

Pay per crawl adds a third option to the relationship between a website and an AI crawler. Until now there were two: allow access with no conditions, or block it with robots.txt. Pay per crawl introduces «allow, but only if paid for».

The technical mechanism is the HTTP status code 402, Payment Required. This code has existed since the earliest HTTP standards, originally intended for some micropayment scheme that never became widespread, and for decades it has barely been used in practice. Pay per crawl brings it back: when a crawler requests a page with this control enabled, the server responds with 402 instead of delivering the content, and attaches the price of that visit in a header. The crawler can repeat the request accepting payment, or give up.

It is not yet a W3C or IETF standard with broad adoption: it is a model, implemented as of this writing mainly by one large web infrastructure provider, and its exact behaviour, meaning headers, signature, currency, depends on how each provider implements it. For a site, this changes the logic that robots.txt made familiar: instead of one fixed rule for everyone, a price list emerges that can differ crawler by crawler.

The typical flow has two variants. In the reactive one, the crawler requests the page with no warning, receives a 402 with the price in a header, and if it accepts, repeats the request adding a header confirming payment. In the proactive one, the crawler already includes a header stating the maximum price it is willing to pay; if the price the site has set does not exceed that, it receives the content without the intermediate 402 step.

The crawler's identity is verified through cryptographic signatures (HTTP Message Signatures with Ed25519 keys), not through the user agent, which can be spoofed. The publisher sets a per-crawl price for the whole domain and decides, crawler by crawler, between allowing it free, charging, or blocking it.

As of this writing, August 2026, Cloudflare is the provider that has publicly implemented this model, in private beta since its announcement on July 1, 2025. On July 1, 2026, it also announced that, starting September 15, 2026, crawlers that combine several purposes at once, for example indexing for search and training models, will be blocked by default on pages that carry ads, unless the site changes that setting. There is no public record of another CDN provider having launched an equivalent implementation as of this writing, so statements about the model in general are, for now, also statements about this one implementation.

The 402 code had been reserved and practically unused since the HTTP protocol was defined. A large infrastructure provider turning it into an active mechanism, and additionally announcing a change to default settings for September 2026, changes the question facing any site with AI crawler traffic: it is no longer just «do I allow or block», but «who decides what happens if I configure nothing».

For a publisher or a site with valuable content, not deciding has a concrete consequence from that date onward: some crawlers will end up blocked by default on ad-supported pages, while others will keep getting in for free, depending on how the infrastructure provider classifies each one. Setting the policy explicitly, instead of leaving the default in place, is the only way for the decision to belong to the site rather than to the provider.

For a ZDS client with its own editorial content or original research, this translates into a concrete task: reviewing, before that date, which AI crawlers currently have access, which ones are worth continuing to allow for free, for example those that cite with a link back, and which should be charged or blocked. Postponing that decision simply hands it to the provider default once the deadline arrives.

• Check which CDN or bot-protection provider the site uses and whether it already offers a payment or blocking control by AI crawler type.
• Don't confuse pay per crawl with a general firewall or block: it only acts on requests identified as AI crawlers that respect the mechanism.
• Set the policy per crawler explicitly (allow, charge, block) instead of leaving the provider's default in place, especially ahead of announced change dates.
• If you turn on a price, also document the terms in robots.txt or in a licensing standard such as RSL, so there is a record outside the server itself.
• Check the server logs periodically to see how many 402 requests are actually happening and from which crawlers, before assuming the mechanism is generating revenue.
• Distinguish between blocking a training crawler and a search crawler: a misconfigured rule can end up blocking traffic from a search engine you actually want to keep.

• Assuming pay per crawl is already an open standard with many providers: as of this writing it is tied mainly to one provider's implementation.
• Thinking that turning on charging automatically blocks every unwanted bot: it only applies to crawlers that recognise the mechanism and choose to pay or give up.
• Not checking that a crawler serving several purposes at once, such as search and training, can end up blocked entirely if only one of those purposes is blocked.
• Leaving the configuration on automatic after a change date announced by the provider, without checking what remains allowed and what ends up blocked.
• Presenting the model as a revenue guarantee: payment depends on the crawler agreeing to pay, it is not automatic.