What Is a Search Engine?

The name describes what the system does: it searches by itself. There is no central registry of the web. The search engine discovers pages on its own by following links and reading sitemaps, and it decides which pages to store and in what order to show them.

A search engine is different from a web directory, where people sort websites by hand, and from an AI assistant that writes an answer. The lines are blurring, though: Google shows AI-generated overviews above its results, and several AI assistants query a search index before they answer.

Google describes its search in three stages and warns that not every page makes it through each one. In the first, crawling, a program called Googlebot downloads pages it finds through links, sitemaps and other sources. In the second, indexing, Google analyses the content, groups near-identical pages and picks one as canonical, the one that may appear in results.

In the third stage, serving results, the search engine selects and ranks pages from the index for each query. According to Google, relevance is determined by hundreds of factors, including the location, language and device of the person searching. That is why two people can see different results for the same query.

Google also makes clear that it does not accept payment to crawl a site more often or to rank it higher. Ads on the results page are sold separately and are labelled as sponsored.

SEO prepares a site for these three stages: the search engine has to be able to crawl it, understand and index its pages and judge them relevant for the right queries. A mistake in the first stage, such as a block in robots.txt, makes the work on the other two pointless.

In Spain, search is highly concentrated. According to StatCounter, which measures visits on the sites in its network, Google accounted for 94% of searches in September 2026, Bing for 3.5% and Yahoo for 1.9%, in rounded figures.

On zds.es we do not distinguish between search engines. The robots.txt, measured on 13 October 2026, has a single block of rules for all crawlers, with no exceptions for Googlebot, Bingbot or AI crawlers: it allows everything except seven internal paths, such as the admin area and the API, and it gives the address of the sitemap.

• Check that robots.txt does not accidentally block pages that should appear in results.
• Submit the sitemap in Google Search Console and in Bing Webmaster Tools.
• Review which pages each search engine has indexed, not just Google.
• Decide explicitly how to handle AI crawlers instead of leaving it to chance.
• Use clear canonical URLs so the search engine picks the right version of each page.
• Measure traffic by search engine to know how much each one matters to your own site.

• Believing that paying for ads improves organic rankings.
• Ignoring Bing Webmaster Tools, which provides its own crawling and indexing data.
• Blocking resources in robots.txt that the search engine needs to understand the page.
• Assuming that a crawled page is also indexed.
• Confusing the average position in Search Console with the position each person sees.