What Is the X-Robots-Tag?

The X-Robots-Tag is a field in the HTTP response with which a server tells search engines what they may do with the file just requested: index it or not, follow its links or not, keep a copy or not. It is the same conversation as the meta robots tag, only held somewhere else.

That somewhere else is the whole difference. The meta tag lives inside the HTML, so it exists only where HTML exists. The header travels ahead of the file, whatever type it is, and is therefore the only route to a PDF, an image or a downloadable file.

It is worth separating it straight away from the other instruction it gets confused with. robots.txt governs whether the crawler may request the file; the X-Robots-Tag governs what it does with it afterwards. Two different moments, and that distinction explains almost every mistake further down.

The X- in the name recalls that it began as a header proposed outside the standard, like so much else from the web's early years. The large search engines honour it today, but it remains a request rather than an order: any client may ignore it without breaking a rule.

From which follows something worth being clear about before going on. Everything this header does happens on the reader's side, not the file's. It does not encrypt it, move it or delete it: it only changes what a search engine decides to do with a copy it already has in front of it.

The practical question is not which of the two instructions is better but which one reaches the file you want to control. On an ordinary page both work and the choice is one of convenience. As soon as the file stops being HTML, only one remains.

The case that turns up most often in technical SEO audits is PDFs: old catalogues, price lists from three years ago, manuals nobody maintains. They are indexed, appear in results with their own title and compete with the pages that do get looked after. There is no tag to put inside them, so control necessarily runs through the header.

Images are similar, though the effect differs: removing an image from the index does not remove the page holding it, it merely stops appearing in image search. And one clarification that saves alarm: the header neither protects nor hides a file, it stays reachable for anyone who has the URL.

A fourth group of files occurs to almost nobody until it shows up in the report: spreadsheets, text documents and archives uploaded "temporarily" to share with a client. They stay, the crawler finds them through a forgotten link, and they end up indexed with their file name as the title.

It is worth looking at from the crawl budget angle: each of those files is requested and processed like any other URL. On a small site it goes unnoticed; on one with thousands of documents, the crawling spent there is crawling that never reaches the pages that matter — visible in the Search Console reports.

The instructions are the same ones the meta tag accepts, and they are worth knowing because the difference between two of them produces very different outcomes. noindex takes the URL out of results; nofollow tells the search engine not to follow the links it contains, which affects internal linking and should not be set out of habit.

Then come the ones governing how the result looks in the SERP: nosnippet suppresses the sample text, max-snippet limits it, noarchive prevents the cached copy and noimageindex leaves that page's images out. They combine, comma-separated, on a single line.

Two more are rarely used and solve specific cases: unavailable_after with a date, meant for content that expires such as a campaign page or an event, and none, which means noindex, nofollow at once. The instruction can also be aimed at one particular search engine by putting its name in front — useful when one of them should be treated differently.

Two of these instructions deserve a warning, because they get set more lightly than they should. nosnippet removes the sample text from the result, and a result without a description gets fewer clicks: the CTR drops even when the position does not move. It makes sense for paid or sensitive content, not as a general rule.

The other is noarchive, which prevents the stored copy. That sounds harmless, but the copy is what some users reach for when the site does not respond, and equally part of the material summarising systems work from. Before setting it, know what you are giving up — as with indexing decisions generally.

It is configured on the web server or in the application that answers, not in the content. In Apache and Nginx you add a rule applying the header to a file pattern; a CMS usually has a setting per content type, and in your own application you add it like any other header.

Checking is the step most often skipped and the only one that proves anything. Request the URL and look at the response headers, not the rendered page: the instruction appears neither on screen nor in the source. Any client that shows the HTTP response will do, and the URL inspection in Search Console additionally shows how Google read it.

And check the real file, not a sample page. Pattern rules fail exactly at the edges: an extension in capitals, a folder left out, a file served from a different subdomain with its own certificate with its own configuration.

A warning about where the rule sits: if a content delivery network or a caching layer stands in front, the header has to survive that step. Some configurations forward only a set of known headers and drop the rest, so the rule exists on the origin server and never reaches the crawler. Check the public URL, not the internal one.

And another about the order of checks. Before looking at the header, make sure the URL answers 200 and not a 301: a redirect carries the whole conversation away, because the headers that count are the final destination's, not the hop's. It is the same slip that makes a rule look broken when the wrong response is being examined.

The first is the classic one and deserves saying plainly: blocking the URL in robots.txt while also setting noindex in the header. The block prevents the request, so the header is never read and the page can keep appearing. To take something out of the index you must let it be crawled.

The second is leaving it in place. A rule from a migration or a test environment copied into production keeps whole sections out of the index for weeks unnoticed; the drop in traffic then gets attributed to something else entirely.

And the third, subtler one: using it where the problem is different. A page that should not exist and ends up as a soft 404 or as thin content is not fixed with noindex, it is fixed by no longer generating it. The header is an instruction to search engines, not a tidy-up of the site's architecture.

A fourth mistake sits not in the configuration but in the expectation: looking for the effect overnight. The instruction only takes hold when the search engine requests the file again, and on a rarely visited URL that can take weeks. An index that has not emptied after forty-eight hours does not mean the rule is failing, it means nothing has been re-checked yet.

And a recommendation that saves arguments: write down which rule was set, where and why. These headers are configured once and read two years later, when whoever set them has gone and nobody dares remove them just in case. One comment line beside the rule keeps a reasoned decision from being treated as superstition.