Sitemap: what it is, XML versus HTML, and why it can't guarantee indexing

The word sitemap covers two distinct tools that share a name but not an audience. The XML sitemap is a file with an .xml extension listing a site's URLs in a machine-readable format, built for search engine crawlers, Googlebot included, so they can discover pages without relying solely on internal links. Each URL can carry extra metadata, most notably the last modification date, which helps a crawler decide what to check first.

The HTML sitemap is the opposite: an ordinary page on the site, visible to any visitor, that links out to the site's main sections or URLs. For years it was common to tuck one into the footer as a navigation safety net, especially on sites with weak menu structure. No crawler treats it specially; to Google it's just another page, with the same internal links as any other.

Today the HTML sitemap has lost most of its reason to exist. A well-designed menu, a working internal search, and careful internal linking cover the same navigational need without a dedicated page. It still earns its keep on very large sites with hundreds of sections, or on projects where accessibility requires a clear path to any page within a few clicks.

When someone in SEO says sitemap without qualifying it, they almost always mean the XML version. It's the one submitted through Search Console, the one referenced in robots.txt, and the one with its own technical rules: size limits, alternative formats, and a dedicated tracking report. The rest of this article focuses on that format, noting differences with the HTML version only where they matter.

Google's official documentation is explicit about what a sitemap does and doesn't do. The guide to building and submitting sitemaps states plainly: "Keep in mind that submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site." This quote was checked directly against Google Search Central's documentation, accessed on 09.08.2026.

The sitemaps overview page repeats the point in different words: "A sitemap helps search engines discover URLs on your site, but it doesn't guarantee that all the items in your sitemap will be crawled and indexed." Google decides independently which URLs it visits and which it adds to its index, based on content quality, internal and external links, and technical signals such as a page's status code or canonical tag.

This distinction explains a common misunderstanding: publishing and submitting a sitemap doesn't make a page show up in Google the next day. A sitemap helps discovery when internal linking is weak or a site is new, but it doesn't substitute for content that is crawlable, indexable, and linked from within the site itself. If a URL listed in the sitemap doesn't get indexed, the cause is almost never the sitemap itself, it's usually the page's quality or its technical indexing status.

The XML sitemap's optional fields reinforce this idea of a hint rather than an instruction. Google ignores priority and changefreq entirely, fields originally meant to signal a URL's relative importance and how often it changes. The only field Google can actually use is lastmod, and only when it's accurate and verifiable: when it reflects a genuine content change, not a timestamp that resets itself every time the file regenerates.

This is exactly where many CMS setups trip up. Plenty of sitemap plugins default lastmod to the date the sitemap was last generated, not the actual content change date. Anyone who doesn't correct that setting manually ends up sending Google a false signal for months, then wonders why the field seems to have stopped having any effect. A quick spot-check usually settles it: if lastmod changes on every single crawl even though nothing on the page has actually been touched, that mismatch is the culprit.

Every sitemap file carries a technical limit set by the protocol itself: a maximum of 50,000 URLs or 50MB uncompressed, whichever comes first. The limit applies to every accepted format, XML, plain text with one URL per line, or the RSS and Atom feeds, though those last two offer far fewer capabilities and don't support fields like lastmod with the same level of detail.

When a site exceeds that limit, the fix isn't forcing everything into one enormous file, it's splitting the URLs across several smaller sitemaps and grouping them under a sitemap index: a file that doesn't list page URLs at all, only the paths to the other sitemaps. Only that index file needs submitting to Google; the crawler follows the internal references down to each individual sitemap.

This structure is common on large e-commerce catalogs, news portals with an extensive archive, or any site with more indexable URLs than a single file's limit allows. It's also good practice to split by content type, one sitemap for products, another for blog posts, another for category pages, even before a site hits the numeric limit: it makes it much easier to spot which block is causing trouble when something breaks.

The 50MB limit always refers to the uncompressed file. It can be served gzip-compressed to save bandwidth, and Google decompresses it before applying the actual limit. That makes it possible to approach the 50,000-URL ceiling even on sites with long URL names, without the compressed file's size becoming a practical problem. In practice, it's worth splitting well before the limit rather than pushing it to the last byte, since an already-large site tends to keep growing on several fronts at once and can cross the line without anyone noticing.

• Include only URLs that return 200, are canonical, and are meant to be indexed.
• Submit the sitemap through the Sitemaps report in Search Console, or reference it in robots.txt if you don't have owner permissions.
• Use lastmod only when it reflects a genuine content change, never as an automatic regeneration timestamp.
• Split the sitemap into multiple files with a sitemap index once you approach 50,000 URLs or 50MB.
• Separate by content type, products, articles, static pages, to isolate problems faster on large sites.
• Check the status in the Sitemaps report regularly: Success, Couldn't fetch, or Has errors.
• Update the sitemap automatically when content is published or removed, not manually and sporadically.
• Gzip-compress the file on large sites to cut transfer size without touching the actual URL limit.
• Skip a dedicated HTML sitemap when the menu and internal linking already cover navigation clearly.

• Leaving noindex URLs in the sitemap: Google gets a contradictory signal and trusts the file a little less.
• Including redirecting URLs, 301 or 302, instead of the final destination URL.
• Listing URLs that return 404 or 500 after a structure change without updating the sitemap.
• Mixing non-canonical URLs in alongside their canonical versions, sending contradictory signals about which version to index.
• Filling in priority and changefreq expecting them to influence crawling: Google ignores both completely.
• Forgetting to regenerate the sitemap after a domain migration or a URL restructure.
• Exceeding the 50,000-URL or 50MB limit in a single file instead of splitting it via a sitemap index.
• Never linking the sitemap in robots.txt or submitting it in Search Console, leaving Google to rely on spontaneous crawling alone.