Skip to content

Glossary Crawl Budget

Crawl Budget: what it actually means and when it matters, according to Google's own docs

Manuel Riveiro Rodriguez
Manuel Riveiro Rodriguez

CEO & Digital Strategist

8 min reviewed on 2026-08-08

Definition

Crawl budget is the number of URLs Googlebot can and wants to crawl on a website within a given timeframe. It is not a switch you flip; it is a practical ceiling that only becomes noticeable on large or very fast-changing sites.

A handful of matches on stone, several already burnt — beside the title Crawl Budget
The matches are counted; some are already spent
On this page 5
  1. The two factors that make up crawl budget
  2. The most repeated exaggeration about crawl budget, corrected against the official source
  3. When crawl budget stops being a theoretical concern
  4. Best practices for crawl budget
  5. Common crawl budget mistakes
In brief

Google itself warns that crawl budget only becomes relevant for sites with more than a million pages that change weekly, or more than ten thousand pages that change daily. For the vast majority of small and medium sites it is not the bottleneck; worrying about it too early distracts from far more urgent indexing problems.

A handful of matches on stone, several already burnt — beside the title Crawl Budget
The matches are counted; some are already spent

The two factors that make up crawl budget

A domain's crawl budget depends on two independent variables Google combines. The first is the crawl rate limit, a technical figure reflecting how many simultaneous requests the server can handle without slowing down. Google calculates it from response times and error rates: a slow server, or one returning a lot of 5xx errors, gets fewer requests, not more, even if thousands of pages are waiting to be crawled.

The second variable is crawl demand, how much interest Google actually has in visiting a site frequently. It depends on page popularity, how often content genuinely changes, and whether Google judges the site overall to deserve attention. A blog publishing one post a month generates different crawl demand than a news portal publishing a hundred articles a day.

Crawl demand also reacts to specific events: a domain change, an HTTP-to-HTTPS migration, or a large-scale URL restructuring can temporarily spike Google's interest in re-checking the whole site, while a site that hasn't published anything new in months typically sees Google gradually cut back how often it comes back. Neither variable can be controlled directly; both only respond indirectly, through better server performance and content that genuinely changes.

These two variables only turn into a real problem when crawling, the stage that precedes indexing, fails to cover all the pages that matter. That is why this article is best read alongside the Indexing glossary entry: without enough crawling, Google never even gets to decide whether a page belongs in the index.

Why the budget is not a fixed number

The most repeated exaggeration about crawl budget, corrected against the official source

SEO communities routinely repeat that you "need to optimize crawl budget" as if it were a universal requirement. Google's own documentation contradicts that directly. The large site owner's guide to managing crawl budget is explicitly aimed at, among other cases, sites with more than a million unique pages that change moderately, roughly weekly, or medium to large sites, more than ten thousand unique pages, whose content changes very quickly, meaning daily.

Google adds a line that tends to get dropped when this guide gets quoted out of context: "If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide." Google also stresses these numbers are "a rough estimate, not exact thresholds." This quote was checked directly against Google's official Search Central documentation, accessed on 08.08.2026, not a secondhand paraphrase.

A related myth worth debunking with the same official source: many people believe the <priority> and <changefreq> fields in an XML sitemap tell Google which pages matter more. Google states outright that it ignores both values. The only field Google can actually use, and only when it is accurate and verifiable, is <lastmod>. A correct sitemap example, as Google documents it:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.yourdomain.com/product/running-shoes.html</loc>
    <lastmod>2026-07-15</lastmod>
  </url>
</urlset>

lastmod only helps when it reflects an actual content change, not when it auto-updates every time the sitemap regenerates while nothing on the page has changed. Google picks up on that pattern and stops trusting the field across the whole domain.

This is exactly where many CMS setups trip up: plenty of sitemap plugins set lastmod to the date the sitemap was last generated by default, instead of the actual content change date. Anyone who doesn't correct that manually ends up sending Google a false signal for months, then wonders why lastmod seems to have stopped having any effect at all.

When crawl budget stops being a theoretical concern

According to Google's own guide, crawl budget becomes a real problem when the Page Indexing report in Search Console shows many URLs with the status "Discovered, currently not indexed," a sign that Google knows about the URLs but is not prioritizing crawling them. This typically shows up on e-commerce sites with faceted navigation, where every filter combination spins off a new URL and generates millions of technically distinct URLs, on news portals publishing thousands of articles a day, and on large sites with long redirect chains or soft 404 pages, pages that return 200 but are effectively empty, wasting crawling on URLs that deliver nothing.

Outside those scenarios, for a corporate site, a blog, or a small online shop with a few hundred to a few thousand URLs, crawl budget is almost never the limiting factor for indexing. If a new page fails to get indexed, the cause is usually content quality, missing internal links, or canonical errors, not Google running out of crawl budget before it can even reach the page.

The Crawl Stats report, tucked inside Search Console settings, exists precisely so you don't have to guess. It shows crawl requests over time, broken down by response code, by file type, by crawl purpose, discovery of new URLs versus refreshing known ones, and by Googlebot type. A sustained spike in 5xx errors or slow responses in that report is a far more reliable signal of a crawl budget problem than any assumption based on site size alone.

It also matters to separate a genuine crawl budget problem from an architecture problem. If Google crawls normally but chooses not to index what it finds, the bottleneck is not crawl budget at all, it is content quality or duplication, and no amount of robots.txt or server tuning fixes that. Before spending time optimizing crawl budget, rule out the more common indexing causes first, covered in the Indexing article. That order saves the most time in practice, since technical crawl budget fixes go nowhere while the real problem sits somewhere else.

Best practices for crawl budget

  • Block filter and internal search parameter URLs in robots.txt when they add no unique content.
  • Fix long chains of 301 redirects and remove soft 404 pages; both waste crawling for no return.
  • Keep server response times low; a fast server gets more requests from Googlebot, not fewer.
  • Use lastmod in the sitemap only when it reflects a genuine content change, never as an automatic timestamp.
  • Consolidate duplicate content with canonical tags or redirects before it spins off thousands of redundant URLs.
  • Check Search Console for URLs marked "Discovered, currently not indexed" to spot real bottlenecks.
  • Point internal links deliberately at the pages that matter most; Google reads those links as a relevance signal.
  • Check the Crawl Stats report on a regular basis, not only once you already suspect something is wrong.

Common crawl budget mistakes

  • Obsessing over crawl budget on a site with a few hundred pages, where it is almost never the real issue.
  • Believing that filling in priority and changefreq in the sitemap affects crawling; Google ignores both completely.
  • Accidentally blocking important pages while trying to save crawling with overly aggressive robots.txt rules.
  • Letting faceted navigation or session parameters generate endless combinations of crawlable URLs unchecked.
  • Ignoring server error rate as a cause of poor crawling, when it should usually be the first suspect.
  • Confusing crawl budget with indexing: having enough crawl budget does not guarantee a URL ends up in the index.
  • Optimizing crawl budget before ruling out content quality or duplication issues, which are usually the real cause.
  • Not checking whether response times or error rates changed after a server migration or hosting switch.
Manuel Riveiro Rodriguez CEO & Digital Strategist

A technical audit covers this and everything else in one pass.

Request an audit

Frequently asked

Do I need to worry about crawl budget on my website?

Only if your site has tens of thousands of URLs or more and changes very frequently, or if new pages take a long time to get crawled. According to Google's own guide, if your pages get crawled the same day they are published, you don't need to worry about this at all.

At what site size does crawl budget start to matter?

Google gives rough reference points, not exact thresholds: more than a million pages with weekly changes, or more than ten thousand pages with daily changes. Below that, it is almost never the limiting factor.

Does the sitemap tell Google which pages are more important?

Not through priority and changefreq, which Google's own documentation says it ignores entirely. The only field that matters is lastmod, and only when it reflects genuine, verifiable content changes. Importance gets signaled to Google indirectly instead, through internal linking and through where a page sits within the sitemap structure itself.

How do I know if crawl budget is actually affecting me?

Check Search Console for how many URLs show as "Discovered, currently not indexed." If that number is high and growing, it signals Google knows about the pages but isn't prioritizing crawling them, a real symptom of a crawl budget limit. It's worth cross-checking the Crawl Stats report too, to see whether the daily request count is actually flat or declining.

What wastes more crawl budget, a slow server or duplicate URLs?

Both, through different mechanisms: a slow server directly reduces how many requests Google allows in total, while duplicate or parameter URLs spread that already-reduced budget across pages that add no extra value. In practice, a large site dealing with both problems at once usually feels the slow-server effect first, since it caps the total available before distribution even comes into play.