What Is a 401 Error (Unauthorized)?

The 401 belongs to the 4xx class of HTTP status codes, the client errors. Its current definition is in RFC 9110 from June 2022, section 15.5.2: the request hasn't been applied because it "lacks valid authentication credentials for the target resource". The same section requires the server to send a WWW-Authenticate header with at least one challenge, meaning the scheme the client can use to authenticate. The RFC gives the line WWW-Authenticate: Basic realm="simple" as an example. Despite the word "Unauthorized" in its name, the code is about who you are, not what you're allowed to do.

A server returns a 401 when a resource requires authentication and the request arrives with no credentials or with credentials it doesn't accept. If nothing was sent, the 401 asks the client to authenticate. If credentials were sent, the RFC says they've been refused, and the client may try again with new ones. In a browser, Basic protection shows up as a box asking for a username and password. With an API, a 401 comes back for example when the access token is missing or no longer valid.

To Google, a 401 is just another 4xx error. Its documentation on status codes, last updated on February 4, 2026, treats all 4xx errors except 429 the same way: Google doesn't index the URL, and if it was already indexed, it's removed from the index. Googlebot never provides credentials, according to the Search Console help, so whatever sits behind a 401 stays out of search. Google also warns that 401 and 403 aren't a way to lower the crawl rate, because 4xx codes other than 429 have no effect on it.

The robots.txt file itself is a separate case. If it returns a 401, Google acts as if no valid robots.txt file existed and assumes there are no crawl restrictions.

A staging site at staging.example.com is protected with a username and password. Visitors see the login box, Googlebot gets a 401 and doesn't index the site. This is the method Google recommends for private content.

A shop pulls its catalogue from an external API. The token expires, the API returns a 401, and the catalogue stops updating until someone renews the token. This 401 has nothing to do with Google.

A third case is a mistake: an access rule meant for the customer area also covers the whole blog folder. In Search Console those pages appear under "Blocked due to unauthorized request (401)" and drop out of the index.

Search Console has its own category in the page indexing report: "Blocked due to unauthorized request (401)". If you want the page indexed, Google's help gives two options: remove the authorization requirement, or let Googlebot in after verifying its identity. To check the error, it suggests visiting the page in incognito mode. Two quick checks:

• Open the URL in an incognito window. If the login box appears, every visitor without a session gets the 401, and so does Google.
• Request only the headers from the command line, for example with curl -I. The response shows the 401 code and the WWW-Authenticate header.

As protection, a 401 does what it promises. Google recommends password protection for confidential or private content and says that this keeps it out of Google Search, or eventually removes it if it already appears. Neither robots.txt nor noindex achieves that alone: a URL disallowed in robots.txt can still appear in results if other pages link to it, and the noindex rule only works if the crawler can fetch the page. The downside is the flip side of the same thing: anything that should rank can't sit behind a 401, because Google doesn't index what it doesn't receive.

Source: RFC 9110, sections 15.5.2, 15.5.4 and 15.5.5; Google, "How HTTP Status Codes Affect Google's Crawlers" (accessed 11.11.2026).

The sources don't draw the line between 401 and 403 in the same place. The Search Console help assumes a 403 follows credentials that were sent, while RFC 9110 allows a 403 for reasons unrelated to credentials. We cover this in the entry on the 403 error.

• Protect the staging site on the server with a username and password, so every request without credentials gets a 401.
• Don't treat robots.txt alone as hiding it: according to Google, a URL disallowed there can still appear in results if other pages link to it.
• Don't stop at noindex or the X-Robots-Tag header: for private content, Google recommends password protection.
• Open the staging site in an incognito window and check that the login box appears.
• After launch, check that the protection hasn't been copied to the live domain: open several URLs in incognito and look at the 401 category in Search Console.
• If a paid area needs to be indexed, don't use a 401: Google documents structured data that marks paywalled content for that case.

• Always send the WWW-Authenticate header with a 401, as RFC 9110 requires.
• Use 401 when authentication is missing and 403 when the server refuses access for another reason.
• Don't use 401 or 403 to slow down Google's crawling; for that, Google points to its guide on reducing the crawl rate.
• Before letting through anything that claims to be Googlebot, check it with a reverse DNS lookup or against the IP lists Google publishes.

• Treating a staging site as hidden because robots.txt blocks it.
• Leaving the protection active on a public section after a launch.
• Serving robots.txt with a 401 and assuming it keeps Google out: Google treats it as if there were no robots.txt.
• Using 401 or 403 to reduce the load from crawling.
• Letting through any request that presents itself as Googlebot without verifying it.

This entry was written in November 2026 with AI assistance. The facts come from RFC 9110, Google's documentation on status codes, robots.txt, noindex and crawler verification, and the Search Console help. It was reviewed before publication and contains no client data.