What Is robots.txt?

The file always lives in the same spot: the root of the domain, reachable at https://yourdomain.com/robots.txt. A crawler that follows the rules checks it before requesting any other URL on the site, so it needs to be reachable without redirects or blocks standing in front of it.

Each rule block starts with User-agent, which names the crawler the following lines apply to: a specific name like Googlebot, or an asterisk to target all of them. The actual directives come next. Disallow blocks a path or an entire directory; an empty line after Disallow, with no path after it, blocks nothing at all. Allow does the opposite and mostly serves to carve out an exception inside a blocked directory, a single file that should still get crawled even though its folder is closed off. The optional Sitemap directive, with the sitemap's full URL, tells the crawler where to find it, and it can appear more than once if there are several sitemaps.

Two wildcards extend what a few lines can express: the asterisk (*) stands in for any sequence of characters within a path, and the dollar sign ($) marks the exact end of a URL. Combined, they can block, say, every URL ending in a specific parameter without listing each one individually.

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?filter=
Allow: /wp-content/uploads/

User-agent: Googlebot-Image
Disallow: /private-photos/

Sitemap: https://yourdomain.com/sitemap.xml

When several rules in the same block apply to the same URL, Google follows the more specific one, the one with the longer path, not whichever rule appears first in the file. That's why a narrow Allow can sit next to a broader Disallow in the same block without contradicting it.

Disallow and Allow paths are case-sensitive, so /Admin/ and /admin/ read as two different rules to a crawler even if the server maps them to the same folder underneath. Lines starting with a hash (#) are comments, and the crawler skips them entirely, useful for noting why a particular rule exists without changing what the file actually does.

This is the single most repeated mix-up in all of technical SEO: robots.txt has no say in what ends up in Google's index. It only decides what the crawler gets to visit in the first place. Those are two separate stages of the same pipeline, and confusing them produces results that look contradictory unless you understand the mechanism underneath.

A concrete example makes it clear. Block a URL with Disallow, and if that URL still picks up a link from another site, Google can index it anyway, based purely on that external signal, without ever having downloaded its content. What shows up in search is an odd-looking listing: the bare URL, no title, no description, sometimes flagged with "no information is available for this page." That happens precisely because the crawler never read the page, so it never got the chance to read any directive sitting inside it either, including a noindex tag that was supposed to keep it out of the index in the first place.

That's the trap a lot of sites fall into: blocking in robots.txt a page that also carries noindex doesn't exclude it from the index, it creates the worst of both outcomes, where neither directive actually takes effect. For noindex to work, the crawler has to be able to get in and read it, which requires exactly the opposite of a block. The article on noindex covers that directive in detail, its two technical variants and when to use each; the practical rule here is enough on its own: robots.txt controls crawler access, noindex controls index inclusion, and reliably excluding a URL requires the second one, not the first.

The mix-up usually happens because both mechanisms tend to get set up at the same moment, during a technical relaunch or a site cleanup, and because on the surface they feel alike: one rule, one file, one effect on visibility. In reality they sit at two different points in the chain. robots.txt decides upfront whether the crawler even gets to knock on the door. noindex decides only afterward, once the crawler has already gone in and read what's on the page.

robots.txt is a public file. Anyone, human or program, can open https://yourdomain.com/robots.txt in a browser and read exactly which paths a site chose to block. That fact alone should reshape how the file gets treated: it's a note taped to the door asking people not to come in, and a note like that stops nobody who's determined to ignore it.

Well-behaved crawlers, Googlebot, Bingbot, and other bots with an interest in keeping a good relationship with the sites they visit, respect that note because it pays off for them long-term. A malicious bot, a pricing scraper, a vulnerability scanner, has no reason to, and often uses robots.txt for the opposite purpose: as a map of which folders are worth checking first, precisely because someone went out of their way to ask that they not be looked at.

That's where the costliest mistake with this file comes from: listing sensitive paths under Disallow, an admin panel, a staging environment, a backup folder, assuming that hides them. The effect is the reverse. Those paths are now published in plain text for anyone reading the file, while a bot that ignores the directive visits them without any real obstacle. Actually protecting something requires authentication, a password, an IP allowlist, not a line in a file anyone can read. The article on crawlers includes a diagram comparing this exact point, blocked isn't the same as gone, worth a look if the distinction still isn't landing.

Google's own documentation states it plainly: robots.txt is not a mechanism for keeping a page out of its search engine. It's meant to manage the load a crawler puts on a server and steer that traffic toward what matters, not to build an access barrier. Treating it like a lock is often what lays the groundwork for the exact kind of data exposure that later gets discovered by accident, when someone finds the file and follows precisely the paths it asked them not to visit.

• Keep the file at the domain root, in plain text, reachable without redirects, so the crawler finds it on the first try.
• Use Disallow only for paths that genuinely add nothing to crawling: internal admin areas, internal search results, filter parameters that multiply URLs without any new content.
• Add a Sitemap line at the end of the file so the crawler can find the sitemap without depending solely on internal links.
• Check the rules with Search Console's robots.txt tester before publishing, especially anything using wildcards, to confirm they do exactly what you think.
• If a URL needs to come out of the index, don't block it with Disallow, use noindex and let robots.txt keep access to that path open.
• Review the file after any major restructuring or migration; a rule inherited from a staging environment can block an entire site without anyone noticing for weeks.

• Blocking an entire site by accident because a rule like Disallow: / stayed active after moving from staging to production.
• Listing sensitive paths under Disallow assuming that protects them, when it actually publishes them for anyone who opens the file.
• Blocking CSS or JavaScript folders the crawler needs to render the page correctly, which can leave Google seeing a broken version of the site.
• Combining Disallow and noindex on the same URL, so the crawler never reads the exclusion tag and the page can keep showing up, without a description, in search results.
• Assuming a URL blocked in robots.txt disappears from Google immediately and permanently, when it can stay indexed if other pages link to it.