What is an AI agent?

When a person asks ChatGPT, Claude, or a Google product to compare prices, book something, or fill out a form, the assistant may visit one or more web pages at that exact moment to complete the task. That visit is carried out by a specifically named component: OpenAI identifies it as ChatGPT-User, Anthropic as Claude-User, and Google as Google-Agent. All three are officially documented by their providers, each with its own user-agent string.

This differs from an AI crawler such as GPTBot, ClaudeBot, or PerplexityBot. A crawler roams the site automatically and continuously, with no person waiting on the result at that instant, and the material it gathers feeds a model's training or builds a search index. An agent does the opposite: it acts once, at the explicit request of one specific user, and the result of that visit flows straight back into that person's conversation.

The boundary matters for whoever runs a website. Blocking a training crawler affects whether content gets cited in the future. Blocking an agent affects whether a real customer can finish, right now, a purchase or a booking they had already started. Training and indexing crawlers already had their own blocking policies before agents existed; these add a third category, real-time action on behalf of one specific person, which didn't exist when those first policies were written.

The flow always starts with a person: they ask the assistant to compare products, check availability, or finish a task. The model decides it needs current data from a specific URL, and the provider's server makes an HTTP request to that page, identified with the corresponding user-agent. That request isn't part of a scheduled crawl queue, it exists because one specific user triggered it at that moment.

The three providers document their behavior toward robots.txt differently. OpenAI states explicitly about ChatGPT-User: "ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply". Anthropic, by contrast, describes Claude-User as an agent that can be blocked just like its other bots, under the same robots.txt rule that would apply to a crawler. Google classifies Google-Agent as a "user-triggered fetcher", a category separate from Googlebot, and its documentation doesn't spell out with the same clarity as Anthropic's whether that category respects robots.txt the same way its regular crawlers do.

This difference between providers is why it doesn't pay to treat "AI agents" as one uniform block. A robots.txt rule that successfully blocks Claude-User doesn't have to produce the same effect on ChatGPT-User, according to OpenAI's own documentation. In practice, this means checking each provider's documentation separately, rather than assuming a single rule or a single blocking plugin covers all three agents equally.

The decision that hinges on this is whether to block these agents at the firewall or in the bot manager, and by what criterion. Blocking "anything that sounds like AI" indiscriminately carries a cost that isn't always visible: if an agent is acting on behalf of a real customer who is trying to buy or book something on your site, that block doesn't stop a model's training, it stops a sale that was already underway.

This also affects measurement. A server that doesn't distinguish agent sessions from regular browser sessions can either count those visits as ordinary human traffic, or lose them entirely if the agent doesn't run JavaScript the way a browser does. Neither gives an accurate picture of how many visits and conversions actually come from an agent acting on a user's behalf.

A concrete example: a travel agency that blanket-blocks any user-agent with "GPT" or "Claude" in the name may be unknowingly blocking a customer who asked their assistant to finish booking a room they had already picked out. The booking doesn't go through, and in the hotel's logs that visit just shows up as one more blocked request, with no indication that a real, in-progress sale sat behind it. Nobody at the agency gets an alert that a booking was just lost, because a blocked request looks the same in the log as any other blocked request.

• Distinguish, in robots.txt and in the bot manager, training crawlers (GPTBot, ClaudeBot) from user agents (ChatGPT-User, Claude-User, Google-Agent).
• Don't block user agents by default if part of your business depends on a customer completing a purchase or booking through them.
• Verify the user-agent string against each provider's official documentation before making a blocking decision, because traffic exists that impersonates an agent without being one.
• Check the server logs to see how many agent visits your site receives and on which specific pages.
• Make sure checkout and forms work without depending on complex JavaScript, because not every agent runs it the way a browser does.
• Update the list of allowed agents every few months: the number of providers with their own agent keeps growing.

• Blanket-blocking anything that sounds like AI at the firewall and losing sales an agent was completing on behalf of a real customer.
• Confusing a user agent with a training crawler and applying the same blocking policy to it.
• Assuming every agent respects robots.txt the same way, when OpenAI explicitly states it may not apply to ChatGPT-User.
• Not verifying the user-agent string, which lets fake traffic pass itself off as a legitimate agent.
• Measuring conversions without separating agent sessions from the rest, which inflates or hides their real impact.