What are AI crawlers

The label lumps different things together, and that is where the problem starts. An AI crawler is not a technical category: it is any automated agent operated by a language model company. What separates them is not the technology but the purpose, and that purpose decides what happens when you block one.

There are three purposes and they are worth naming separately. The first is training: fetching text to feed future models. The second is the provider's own search index, the one that lets an assistant return links when it answers. The third is live retrieval, when the user pastes a URL or asks something that forces the assistant to go and fetch the page at that very moment.

Each provider uses a distinct agent for each purpose and documents them separately. OpenAI separates GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic separates ClaudeBot, Claude-SearchBot and Claude-User. Perplexity separates PerplexityBot and Perplexity-User. Google breaks the pattern: it has no agent of its own on this list. It crawls with Googlebot and offers Google-Extended as a control token, not as a bot. Google documents that it has no user agent string of its own and that fetching is still done by its existing agents, which is why that name will never show up in an access log. All it decides is whether already crawled content may be used for Gemini training and grounding.

Treating all of this as one block, with a single rule that covers them all, is the cause of most bad decisions in this area.

Control runs through the same old file: robots.txt at the root of the domain, with one block per agent. The name written after User-agent is the token each provider publishes, and a rule aimed at one token does not affect the others.

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Each line does something specific. Blocking GPTBot signals, per OpenAI's own documentation, that the content should not be used to train generative models. Allowing OAI-SearchBot is the route OpenAI recommends so the site can appear in ChatGPT's answers with links, and robots.txt changes take around 24 hours to take effect in that system. Blocking Google-Extended removes the content from Gemini training and grounding, and Google documents that this token affects neither inclusion in its search engine nor ranking; it is worth remembering that here the rule stops no download at all, because the token issues no requests of its own. Anthropic additionally accepts Crawl-delay for ClaudeBot and publishes its IP ranges for verifying requests.

One exception breaks the pattern. Perplexity documents that its Perplexity-User agent, the one that acts when somebody asks a question, generally ignores robots.txt rules, on the grounds that a person initiated the request. OpenAI uses a similar formula for ChatGPT-User: because these are user-initiated actions, robots.txt rules may not apply. A blocking directive therefore stops systematic crawling, but it does not guarantee that nobody retrieves the page live.

The decision hanging on this has two sides, and anyone telling only one of them is selling something. Blocking the training agents takes the site out of the material from which future models are built. Blocking the index agents does something else: it removes the site from the set of pages an assistant can cite and link to when it answers. You can want the first without wanting the second, and that separation is precisely what each provider's documentation allows.

A paywalled publisher or a house negotiating licences has reasons to close off training. A shop or a service company that wants to be cited in answers usually has the opposite interest regarding the index. The most expensive wrong rule is the one that uses a wildcard to block everything starting with the provider's name and shuts out the search crawler along the way.

A third factor remains, resource consumption. On large sites these requests add up and can raise bandwidth costs without producing visits. That is measured in the server access logs before deciding, not estimated from memory. The conversation about what to block starts with knowing who is requesting what on your domain, and at what volume.

• Write one robots.txt block per token and per purpose; avoid wildcards that conflate the training agent with the search agent.
• Before blocking anything, filter thirty days of access logs by each token and work out requests, bytes served and pages affected.
• Verify the agent's identity against the IP range lists the providers publish; the user agent field is written by whoever sends the request and is forged in a second.
• Record the decision and its reason next to the file, with a date, so whoever reviews it a year from now knows what it was meant to achieve.
• If you need a block that does not depend on the agent's goodwill, apply it at the server or at the CDN provider, not only in robots.txt.
• Review the file every quarter: providers add new tokens, and a rule written two years ago no longer covers what you think it covers.

• Using a wildcard User-agent with a full Disallow to hold back AI, which also drops the site out of the classic search engines.
• Blocking Google-Extended expecting to disappear from AI Overviews. That token covers Gemini training and grounding, while search answers are fed by the index Googlebot fills. To leave those surfaces, Google announced a dedicated Search Console setting on 3 June 2026.
• Assuming robots.txt stops live retrieval, when Perplexity and OpenAI document that their user agents may not apply those rules.
• Identifying traffic by the agent name alone, without checking the source IP address against the provider's list.
• Copying a block list found on a forum, with obsolete or invented tokens, and never reviewing it again.