What Is Crawling? How Search Engines Scan Your Website

A crawl starts before the bot ever requests your page. Googlebot first builds a list of candidate URLs from three sources: links found on pages it has already crawled, addresses listed in your XML sitemap, and URLs it already knows from earlier visits. That list feeds a crawl queue, and the order in which it gets worked through depends on how much crawl budget your domain has and how high a priority each URL is estimated to carry.

For a brand-new domain, that process starts differently. Google has no prior URL to work from, so it needs a first entry point. That's usually an external link from a page Google already visits, a sitemap submitted manually through Search Console, or a single crawl request via the URL Inspection tool. Without any of the three, a domain can stay completely off the crawler's radar for a long time.

When a URL's turn comes up, the crawler sends an HTTP request and downloads the HTML. If the page's main content depends on JavaScript, that first HTML pass usually isn't enough. Google flags the page for a second round, where a headless Chromium, the Web Rendering Service, runs the code and builds the final DOM the way a browser would. That second pass often lands days later, which means a page loaded with JavaScript-injected content can take longer before Google sees it in full.

Every crawled page hands back new links, which rejoin the queue as fresh candidates, and the cycle starts again. A URL doesn't drop off the list after its first visit. It gets a slot in a recrawl schedule that decides when Google checks back.

The two terms get mixed up because they share a root, but they describe different things. A crawler is the program: a bot like Googlebot or Bingbot, with its own user agent, its own behavior rules, and its own way of reading robots.txt. Crawling is what that program does: the process of discovering, visiting, and revisiting pages over time.

The distinction shows up in the questions each one answers. "Which bots visit my site, and how do I identify them?" is a question about the crawler. "How often does Google come back to this section, and why does it take so long to find new pages?" is a question about crawling. For a rundown of the most active crawlers in 2026 and how they work, that's covered in the dedicated crawler article. This one stays with the process behind them.

That split affects the fix, too. Raising how often Google revisits a section means working on internal links and fresh content, not bot settings. Blocking an aggressive, unfamiliar bot that only claims to be a crawler is a different problem, and the answer lives in the crawler article: which user agent it actually uses, and how to block it reliably through robots.txt or by IP.

Google doesn't visit every page on a domain at the same rate, and there's no fixed schedule behind it. Two factors decide when a page gets revisited: its estimated importance (how many internal and external links point to it, how much traffic it drives, whether it sits in the sitemap) and how often it has changed in the past. A news homepage updated hourly can get visited several times a day. A privacy policy that hasn't moved in two years can sit for weeks between visits.

E-commerce makes the difference concrete. A product page where stock and price change daily tends to get frequent visits, because Google has learned there's something worth checking. An "About us" page that hasn't moved in three years sits at the other end: Google knows it, has it indexed, but rarely bothers revisiting because experience says nothing there is going to change.

Seasonal campaign pages follow the same pattern in reverse. A campaign page that's only live two weeks a year loses visit frequency during the off-season, because Google finds little changing there during that stretch. That's not a flaw in the system, it's the logical result of the same rule: importance and change frequency get reassessed continuously, not set once and left alone.

That has a practical consequence. Publish new content regularly in a section, and Google learns that section changes, which can raise the visit frequency over time. The reverse holds too. A section full of URLs that never change, or that keep returning errors, gets fewer visits because Google has learned that checking rarely turns up anything new.

The server itself plays a role here. When Google detects slow response times or repeated 5xx errors, it throttles the rate of requests to avoid overloading the site, something it documents directly in its crawl budget guidance.

The most direct way to see what Google does on your site is the server logs. Every Googlebot request leaves a line with the date, the URL requested, the status code returned, and the user agent. Filter those lines by user agent and verify the IP against Google's published ranges, and you can see exactly what gets crawled, how often, and with what result, independent of whatever Google's own tools choose to surface.

A typical Googlebot log line looks something like this: the requesting IP, the date, the method and URL requested, the status code, and the user-agent string, for example GET /category/product 200 followed by "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)". The user agent alone proves nothing; it's trivial to spoof. Reliable verification means a reverse DNS lookup of the requesting IP, checking that the resolved domain belongs to googlebot.com or google.com, then a forward lookup back to confirm it points to the same IP.

Search Console offers a more convenient, if less complete, view through the Crawl Stats report (Settings > Crawl Stats). It shows total requests over the last 90 days, broken down by response code, by file type, and by whether the visit was for discovery of a new URL or a refresh of one already known. It also separates requests by Googlebot type, so desktop crawling can be told apart from mobile or image crawling.

Neither source replaces the other. Logs show the line-by-line detail, the Search Console report shows the trend and makes spikes or drop-offs visible without combing through thousands of lines by hand.

A sudden drop in requests on the Crawl Stats report is often the first visible warning sign of a technical problem, showing up before it ever shows in rankings or traffic. A broken deployment, a misconfigured robots.txt rule, or an overloaded server typically appears there before it appears in analytics.

• Keep the XML sitemap current, listing only URLs you actually want indexed.
• Check robots.txt regularly, so no section gets blocked by accident.
• Link to new pages internally. It's the fastest way for Google to find them at all.
• Watch server response time. A slow site gets fewer crawl visits.
• Fix redirect chains and broken links. Every extra hop spends crawl budget.
• Check the Crawl Stats report in Search Console at least once a month.

• Confusing "it's in the sitemap" with "it's crawled." A sitemap is a suggestion, not an order.
• Blocking pages in robots.txt that are also supposed to carry a noindex tag. Blocked crawling means Google never sees that tag.
• Expecting a new page to get crawled the day it's published. Without internal links pointing to it, that can take weeks.
• Assuming more crawling is always better. Budget wasted on duplicate parameter URLs is budget the pages that matter don't get.
• Treating the "crawled pages" number in Search Console as the same thing as "indexed pages." They're two different figures.