What Is Googlebot?

The two types differ in the user agent of the HTTP request, but in robots.txt they obey the same token, "Googlebot". You therefore cannot write a rule for the mobile crawler alone or the desktop crawler alone. Because Google indexes mainly the mobile version of most sites, most requests come from Googlebot Smartphone. Mobile-first indexing has its own entry in this glossary.

Google divides its crawlers and fetchers into three groups: - Common crawlers such as Googlebot. On automatic crawls they always respect robots.txt. The same group includes Googlebot-Image, Googlebot-Video, Storebot-Google for Google Shopping and Google-InspectionTool, which is used by testing tools such as the Rich Results Test and URL Inspection in Search Console. Googlebot-News exists only as a robots.txt token and crawls with Googlebot's user agents. - Special-case crawlers such as AdsBot, which Google Ads uses to check ad quality on web pages. They work under an agreement between the site and the Google product and may not follow robots.txt. AdsBot, for example, ignores rules for the global user agent (*). - User-triggered fetchers such as Google Site Verifier. Because a person requests the fetch, they ignore robots.txt.

Google-Extended is not a crawler of its own either. It is a robots.txt token with which a site decides whether content Google crawls may be used to train Gemini models. It has its own entry in this glossary.

Googlebot discovers new URLs mainly through links in pages it has already crawled. According to Google, it should not access most sites more than once every few seconds on average. For Google Search it fetches the first 2 MB of each supported file type and the first 64 MB of a PDF, measured uncompressed. Each resource the page needs for rendering, such as CSS or JavaScript, is fetched separately under the same limit. Anything beyond the cutoff is not considered for indexing.

Google's crawlers egress mainly from IP addresses in the United States and use HTTP/1.1 by default. They can crawl over HTTP/2, which saves resources, but Google says it brings no ranking benefit.

The user agent can be spoofed, and many crawlers pose as Googlebot. Google describes two ways to check that a visit is genuine: - Reverse DNS: look up the hostname for the IP that made the request, check that it ends in googlebot.com, google.com or googleusercontent.com, then resolve that name to confirm it returns the same IP. For common crawlers the name follows the pattern crawl-…googlebot.com or geo-crawl-…geo.googlebot.com. - IP ranges: match the IP against the JSON lists that Google publishes. The list for common crawlers is common-crawlers.json. The old address, googlebot.json, redirects to it (checked on 17.10.2026).

Reverse DNS is enough for one-off checks. For many IPs, Google recommends automatic matching against the lists. Crawling in general and how to analyse it have their own entry in this glossary.

Blocking Googlebot affects Google Search, including Discover, and other products such as Google Images, Video mode and Google News. A block in robots.txt prevents crawling but not indexing: the URL can still appear in results. To keep a page out of the index, Google points to noindex. To stop anyone, crawler or user, from reaching it, another measure such as password protection.

If Googlebot overloads the server, Google recommends returning a 500, 503 or 429 status code instead of 200 for a short time, a few hours or one to two days. Crawling then drops across the whole hostname. If the errors persist on the same URL for several days, Google may drop it from the index.

Before blocking a supposed Googlebot, check it. Google warns that its user agent is often spoofed, and blocking the genuine IPs by mistake takes the site out of crawling.

• Check any visit claiming to be Googlebot with reverse DNS or common-crawlers.json before blocking it or drawing conclusions from logs.
• Give the mobile version the same content as the desktop version, because most requests come from Googlebot Smartphone.
• Keep the HTML and each required resource under 2 MB uncompressed, because Googlebot ignores anything beyond that.
• Use noindex, not robots.txt, when a page should not appear in Google.
• Send ETag in responses: Google recommends it over Last-Modified for its crawlers' HTTP caching.

• Trusting the user agent: anyone can write "Googlebot" into a request.
• Trying to separate Googlebot Smartphone and Googlebot Desktop in robots.txt, when both obey the same token.
• Blocking a page in robots.txt to remove it from results.
• Returning 503 for days to slow crawling, at the risk of Google dropping the URLs.
• Assuming all Google crawlers treat robots.txt the same way: special-case crawlers may not follow it and user-triggered fetchers do not.