What is Google-Extended

The first confusion worth clearing up is one of category. Google-Extended does not download pages. There is no robot by that name crawling the web, and no line containing that string will ever appear in your server logs. The one doing the downloading is still Googlebot, behaving exactly as it always has. What the token does is govern the later use of content that Google already holds in its index.

That difference explains why the rule is written in robots.txt even though it regulates no crawling at all. The file is used as the place where a publisher records a preference, and Google consults it when deciding whether that material may feed certain generative products. Google's own documentation captures the nuance: the token is used in a control capacity, not as a request.

The control appeared on 28 September 2023, when the affected products were still called Bard and Vertex AI. The commercial names later moved to the Gemini family, but the token kept its own, which is why it reads as uninformative today. It is best understood as «extended use of what has already been crawled» rather than «a second Google crawler», because nearly every badly written rule seen in production grows out of that mistaken reading.

The token covers two specific uses. The first is training future generations of Gemini models, including those powering the Gemini apps and the Vertex AI API. The second is grounding, meaning the delivery of content from the search index to the model at the moment someone asks a question inside those products. A Disallow: / rule under the token withdraws the site from both uses at once, with no way to separate them.

What falls outside its scope is precisely what clients ask about. The generative answers appearing inside Search, both AI Overviews and AI Mode, are built from the ordinary index that Googlebot fills, and therefore do not obey this rule. Google states it plainly: the token neither affects the site's inclusion in Search nor is used as a ranking signal.

For nearly three years the consequence was awkward. Anyone wanting out of the generative answers inside Search could only achieve it by restricting Googlebot or limiting snippets with nosnippet, data-nosnippet and max-snippet, measures paid for in classic visibility. On 3 June 2026 Google announced a dedicated setting in Search Console, under Settings, called «Search generative AI control». It offers two options for the property, include or exclude, with inclusion as the default, and it allows a site to be withdrawn from AI Overviews, AI Mode and the generative features in Discover without touching its position in Search. The rollout started with a small group of properties in the United Kingdom before extending to the rest.

The decision that hangs on this token is narrower than its reputation suggests. Blocking it serves one purpose: preventing your own content from contributing to the Gemini models and to the answers those models generate inside Google's applications. It is a decision about content use, close to licensing and intellectual property, and not a visibility lever.

Anyone blocking it in the hope of disappearing from the generative answers inside Search achieves nothing, and in exchange gives up having the brand cited and linked on the Gemini surface. The cost is paid in full and the objective is not reached. That misunderstanding is the main reason a client asks about this token, and it deserves a clear answer before anyone touches the file.

For a publisher with licensable content the reading changes. There, blocking works as a negotiating position, and several publishers maintain it while discussing usage agreements. For a business that lives on capturing demand, by contrast, the usual answer is to allow it and work on the citability of the pages.

Since June 2026 there is also a second lever, independent of the first. Each is best decided on its own, because they answer different questions: one about who may use the content, the other about where it appears.

• Declare a dedicated group with the token spelled exactly, without inheriting wildcards from other blocks in the file and without mixing it into the Googlebot rules.
• Decide the two questions separately: use of the content in the models, which is settled in robots.txt, and presence in the generative answers inside Search, which is settled in Search Console.
• Do not look for visits from the token in your server logs. The rule is verified by reading the published file and using the robots.txt tester, never through log file analysis.
• Record the date and the reason for the decision in writing, because Google's product names have already changed once around this very token.
• Review the Search Console setting once it reaches the property, since the rollout is gradual and not every account has it yet.
• Before blocking anything, measure what traffic and what mentions arrive today from AI surfaces, so you know the real price of giving them up.

• Blocking the token intending to get out of AI Overviews. Those answers feed on the ordinary index and stay exactly the same after the change.
• Assuming the rule also affects OpenAI, Anthropic or Perplexity. Every provider has its own agents and its own rules.
• Concluding the rule does not work because the token never shows up in the logs. It never will, because it sends no requests.
• Copying someone else's robots.txt block that contains a broad wildcard and ends up restricting Googlebot as well.
• Confusing the Search Console exclusion with a noindex. The former withdraws the site from generative answers and leaves classic Search untouched.