Skip to content

Glossary Indexing

Indexing on Google: what it means and how it differs from crawling and ranking

Manuel Riveiro Rodriguez
Manuel Riveiro Rodriguez

CEO & Digital Strategist

8 min reviewed on 2026-08-08

Definition

Indexing is the process by which Google analyzes a page it has already crawled and stores it in its index, the database used to build search results. Without indexing, a page cannot appear on Google, no matter how well it is otherwise optimized.

A brass stamp resting on a stack of blank paper — beside the title Indexing
Without the stamp the page exists but does not count
On this page 5
  1. How a page actually becomes visible on Google
  2. Crawled, indexed and ranked are not the same thing
  3. When indexing turns into a real problem
  4. Best practices for indexing
  5. Common indexing mistakes
In brief

Crawling, indexing and ranking are three separate stages. A URL can be crawled without being indexed, and indexed without ranking for a single keyword that anyone searches. Mixing up these three terms is the most common reason SEOs misdiagnose a "we don't show up on Google" problem.

A brass stamp resting on a stack of blank paper — beside the title Indexing
Without the stamp the page exists but does not count

How a page actually becomes visible on Google

Before a page can compete for organic traffic, it moves through three distinct stages. First, Googlebot discovers and visits the URL: it requests the HTML, usually renders it with a Chromium-based engine to execute JavaScript, and follows the links it finds. This stage is called crawling, and it is a hard prerequisite for everything that follows. If Googlebot cannot reach a URL, that URL never makes it to indexing. We cover crawling in depth in the Crawl Budget glossary entry, worth reading alongside this one since the two processes sit right next to each other.

Once a page is crawled, indexing follows: Google analyzes the content, heading structure, internal links and structured data, then decides whether the URL earns a place in the index. This decision is not automatic. Google can crawl a page without any issue and still leave it out of the index, for instance when the content is nearly identical to another URL on the same site or elsewhere, when the text adds little independent value, or when technical signals conflict, such as a canonical tag pointing somewhere else.

Only an indexed page enters the third stage, ranking, where Google decides which queries it can show up for and at what position. This is where most site owners get caught off guard: being indexed does not mean being visible. A page can sit in Google's index for months and still never receive a single organic visit, because it never surfaces near the top for any query that actually matters.

This order cannot be shortcut. There is no way to make a page rank without it first being crawled and indexed, and no trick that forces indexing without Googlebot having been able to visit the page first. That is why, when visibility looks broken, the right move is checking backwards first: is the page indexed at all? Was it even crawled? Only once both answers are yes does a ranking problem, in the strict sense, become the correct diagnosis.

Crawled, indexed and ranked are not the same thing

This is the most common misreading in technical SEO. A "crawled" page only means Googlebot visited the URL. An "indexed" page means that visit ended with the URL being stored in the index. A page that "ranks" additionally shows up for specific queries and receives real clicks. A page can sit in exactly one of these three states without being in the other two.

The fastest way to check whether a specific URL is indexed is typing site:yourdomain.com/exact-path into Google search. If a result appears, the URL is indexed, regardless of whether it drives traffic. The Page Indexing report in Google Search Console, along with the URL Inspection tool, gives the same information more reliably.

Real control over this chain sits in two mechanisms that get confused constantly: robots.txt and the noindex tag. This mix-up causes more technical audit errors than almost anything else, so it is worth working through actual code rather than staying at the theory level.

robots.txt operates at the crawling stage: it tells Googlebot which paths not to visit. A complete example:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?filter=
Allow: /admin/login.html

Sitemap: https://www.yourdomain.com/sitemap.xml

What many people miss, and what Google's own documentation states outright, is that blocking a URL in robots.txt does not guarantee it stays out of the index. If other pages link to that URL, Google can still index it even though it never read the content, and show it in results without a description, just the bare URL. It is a textbook mistake: someone trying to get a page out of Google by blocking it in robots.txt achieves the exact opposite.

The correct tool for removing a page from the index is the noindex tag. It operates at the indexing stage, not the crawling stage. Example in the HTML <head>:

<meta name="robots" content="noindex, follow">

Here is the trap that catches people most often: noindex only works if Googlebot is allowed to crawl the page. If that same URL is also blocked by robots.txt, Google never reads the noindex tag, because it never downloads the HTML in the first place, so the instruction has no effect at all. The rule that sums up this whole section: robots.txt prevents crawling; noindex prevents indexing despite crawling being allowed. To deindex a URL, that URL has to stay crawlable.

Crawling, indexing, ranking — and where each one fails

When indexing turns into a real problem

The most underrated risk is not missing pages in the index, it is too many low-value pages ending up there. This is known as index bloat: when Google indexes thousands of low-value URLs, such as internal filter results, empty pagination pages, duplicates created by session parameters, or forgotten drafts, the index fills up with noise. That dilutes the domain's perceived relevance and hurts crawl efficiency, since Google spreads its attention across thousands of irrelevant URLs instead of concentrating it on the pages that actually matter.

The classic index bloat scenario shows up on e-commerce sites with faceted navigation, where every combination of size, color and price filter generates a separate URL, or on sites with tracking parameters (?utm_source=) that Google treats as distinct URLs whenever no canonical tag says otherwise. The fix is not always noindex; often a canonical pointing to the clean URL, combined with clear robots.txt rules for parameter paths, solves the problem at the root.

Google documents the difference between robots.txt and noindex precisely in its Search Central, and states explicitly that robots.txt "is not a mechanism for keeping a web page out of Google." That is a direct quote from Google's own documentation, not an interpretation, and it is worth keeping in mind before touching any robots.txt file with deindexing in mind.

Auditing index bloat in an orderly way works best as a three-step process. First, export every indexed URL from the Page Indexing report in Search Console and group them by pattern, for example everything containing ?filter= or every pagination page. Second, decide the right treatment per group: a canonical pointing to the clean version when content is nearly identical, noindex when a page has one-off value but should not compete for indexing, or a 301 redirect when the page no longer serves any purpose. Third, allow four to six weeks before checking the report again, since the effect is not immediate; it depends on Googlebot recrawling each affected URL.

Best practices for indexing

  • Submit and keep the XML sitemap current in Google Search Console; it is the most direct way for Google to discover new URLs.
  • Use noindex, never robots.txt, whenever the actual goal is removing a URL from the index.
  • Check the Page Indexing report in Search Console at least monthly, not only when something visibly breaks.
  • Add self-referencing canonical tags on every page, and canonical parameter URLs to their clean version.
  • Consolidate or 301-redirect near-duplicate content instead of letting it compete for indexing against itself.
  • Make sure no important page ends up orphaned; without internal links pointing to it, Googlebot revisits it less often.
  • After a major site change, confirm indexing status with the URL Inspection tool, not just a site: search, for key pages.

Common indexing mistakes

  • Blocking a URL in robots.txt expecting that to deindex it; it can still show up in results without a description.
  • Combining a robots.txt block with noindex on the same URL, without realizing the block cancels out the tag.
  • Confusing "indexed" with "ranking": these are two separate states and need to be measured separately.
  • Letting faceted navigation or tracking parameters spin off thousands of indexable URLs unchecked.
  • Forgetting to remove the noindex tag from pages that were in maintenance mode and later went back live.
  • Assuming more indexed pages is always better, without distinguishing useful content from redundant content.
Manuel Riveiro Rodriguez CEO & Digital Strategist

A technical audit covers this and everything else in one pass.

Request an audit

Frequently asked

How long does it take Google to index a new page?

There is no fixed timeline. On domains with strong authority and frequent crawling it can happen within hours; on new or rarely visited domains it can take several weeks. Submitting the URL manually through Search Console usually speeds up the first crawl, but it does not guarantee indexing.

Why does Google crawl a page but not index it?

The most common reasons are content that is nearly identical to another URL, content too thin to add independent value, conflicting technical signals such as a canonical tag pointing elsewhere, or Google simply not prioritizing that URL relative to other pages on the same domain.

Can a page be indexed and still get no traffic?

Yes, and it happens more often than people expect. Being indexed only means Google stored the page in its database. Whether it drives visits depends on whether it ranks near the top for queries with real search volume, which you check with Search Console data, not just a site: query.

Does robots.txt stop a page from being indexed?

That is not what it does. robots.txt blocks crawling, not indexing, and Google can still index a blocked URL if other pages link to it. To actually remove a page from the index you need noindex, and the page has to remain crawlable for Google to read that tag at all.

What is index bloat and why should I care?

Index bloat is the buildup of low-value URLs in Google's index: filters, duplicates, empty pagination pages. It rarely triggers a direct penalty, but it dilutes the domain's perceived quality and worsens how crawl attention gets distributed, which can delay indexing of new pages that actually matter.