What Is a Crawler?

A crawler, also called a web crawler, spider, or indexing bot, is a program that visits web pages automatically and systematically. It starts with a list of known URLs, called the seed list, opens each one, reads the content, and detects the links it contains. From there it follows those links to new pages, and repeats the process without human involvement.

The term is sometimes used as a synonym for "bot," but not every bot is a crawler. A crawler explores and discovers content broadly, while a scraper pulls specific data from a page it already knows about to reuse elsewhere. Google itself uses "crawler" as the official term in Search Console and in its developer documentation.

A crawler theoretically travels the entire World Wide Web, though in practice each operator decides how much of it to visit and how often. That exploration is what a site's appearance in a search engine, or its citation by an AI system, depends on, which is why the crawler is one of the first concepts in any Technical SEO foundation.

Not every crawler serves the same purpose. Some index content for a classic search engine, others use it to train AI models, and others crawl it so it can be cited in an answer generated on the spot. The table below covers the five most common and what each one is for.

The most common mistake is treating every AI crawler as the same thing. GPTBot and Google-Extended exist to train models, and blocking them doesn't remove a site from any search engine. PerplexityBot is different: it feeds Perplexity's search product directly, so blocking it can actually pull a site out of its answers.

The process starts with a seed list of URLs, pulled from sitemaps, external links, or previous crawls. The crawler visits each URL, downloads the HTML, and, if it supports it, runs the page's JavaScript before reading the result. Googlebot has rendered JavaScript by default for years, but simpler or older crawlers only read the initial HTML, so a site that depends entirely on JavaScript to display its content can stay invisible to them.

Before requesting a page, a well-behaved crawler checks the domain's robots.txt file and follows the Disallow rules and user-agent-specific directives it finds there, along with any stated crawl-delay. Not every crawler follows those rules to the letter: according to Google's own documentation, different crawlers interpret the syntax differently.

User-agent: *
Disallow: /admin/
Disallow: /*?sort=
Allow: /wp-content/uploads/

Sitemap: https://zds.es/sitemap.xml

Every link a crawler finds on a page is a clue about what to visit next, and also a signal used to calculate the PageRank of the destination page, unless the link carries a nofollow attribute instead of dofollow. Beyond the HTML, a modern crawler can also read a page's structured data to better understand what it's about. Pages it manages to read successfully move on to the indexing stage; pages that return a 400 error or a 500 error get set aside for now, and pages that no longer exist and return a 404 error eventually drop out of the index.

Google sets aside a limited crawl budget for each domain: a limited number of requests it's willing to make within a given period, based on page quality, server performance, and the site's popularity. On a small site, that limit barely registers. On a site with tens of thousands of URLs, especially one that generates many variants through filters or parameters, spending that budget on irrelevant pages leaves less room for the pages that actually matter.

A large share of organic search depends on this: a page the crawler never visits can't be indexed, and a page that isn't indexed can't show up in any result. Something similar applies to AI crawlers, with a nuance: blocking the one that feeds an answer product cuts visibility right there, while blocking a training crawler only affects future models, not the site's current position in any search engine.

Deciding which crawlers get in and which don't is, at bottom, as much a business decision as a technical one.

• Keep the robots.txt file up to date and test it after every major structural change, so entire sections of the site don't get blocked by mistake.
• Publish an up-to-date sitemap.xml so the crawler can discover new pages without relying solely on internal links.
• Avoid long chains of internal redirects, since every extra hop spends part of the crawl budget without delivering new content.
• Check that the CSS and JavaScript resources needed to render the page aren't accidentally blocked in robots.txt.
• Review server logs to see which crawlers actually visit the site, rather than assuming based only on what robots.txt says.
• Decide deliberately, not by default, whether to allow access to AI training crawlers like GPTBot.

• Assuming that blocking a URL in robots.txt removes it from Google: it can still appear in results, without a description, if another site links to it.
• Accidentally blocking the CSS or JavaScript files the crawler needs to render the page correctly.
• Treating every AI crawler as the same, when some train models and others feed a live answer product.
• Confusing a crawler with a scraper: two programs with different purposes, even if both access the same pages.
• Ignoring crawl budget on large sites with thousands of filter-generated URLs, leaving important pages unvisited.