What Is a 429 Error (Too Many Requests)?

The 429 was not part of the early HTTP specifications. RFC 6585, "Additional HTTP Status Codes", published by the IETF in April 2012, introduced it together with codes 428, 431 and 511. The mechanism it describes is rate limiting: capping how many requests a server accepts from each client. RFC 9110 from 2022 does not redefine the 429, so RFC 6585 is still the reference.

Something counts each client's requests and, once the limit is passed, stops serving the page and answers 429 until the waiting time is over. That something can be the application itself, an API gateway or a CDN with a firewall.

The RFC leaves open how the client is identified and how requests are counted. The limit can apply per resource, across the whole server or across a set of servers, and the client can be recognised by its credentials or by a cookie. According to MDN, limits are most often based on the client's IP address. Two more rules from the RFC: a 429 response must not be cached, and servers are not required to use it at all; under attack, dropping the connection can make more sense than answering every request.

For Google, the 429 is the exception among client errors. Its status code documentation treats all 4xx codes the same, except 429: with a 404 or a 403, the URL is not indexed, and if it was indexed it gets removed. A 429, by contrast, signals an overloaded server, and "it's considered a server error". That gives it the effects of a 5xx:

• Google's crawlers slow down, and according to the guide to reducing crawl rate the slowdown covers the whole hostname, including URLs that respond normally.
• Indexed URLs are kept, "but eventually dropped" if the error continues.
• The crawl rate goes back up once the errors drop.

Google recommends 429, together with 500 and 503, to slow its crawling for a few hours or one to two days. It advises against 403 and 404 for that purpose.

A 429 on the robots.txt file follows its own rules. Google's robots.txt documentation treats any 4xx as if the file did not exist, except 429. Search Console Help on the Crawl Stats report lists "HTTP 429/5XX" among the unsuccessful robots.txt responses. A 429 there counts as a connection problem.

For AI crawlers, the documentation is missing. The crawler pages of OpenAI, Anthropic and Perplexity, accessed 12.11.2026, do not say how their bots react to a 429 or to Retry-After. Anthropic does support the Crawl-delay directive in robots.txt. To control them, you have robots.txt and, if you don't want to rely on the crawler's goodwill, blocking at the server. And pay per crawl does not use 429 at all: it answers with a 402 and asks for payment.

RFC 6585 gives this example itself, for a site that allows 50 requests per hour for each logged-in user:

HTTP/1.1 429 Too Many Requests Content-Type: text/html Retry-After: 3600

The 3600 is in seconds, so one hour.

A CDN rule meant for scrapers can also catch Googlebot if it crawls many filter URLs in a short time. If the rule answers 429, Google crawls the whole host more slowly. If it answers 403, Google concludes those URLs do not exist.

To see a URL's status code, a single terminal request is enough: curl -I https://www.example.com/. The first line shows the code, followed by Retry-After if the server sends it. If the limit depends on IP address or user agent, test from the same origin as the affected client.

In Google Search Console, the Crawl Stats report shows the responses Google received by type, plus host status, including robots.txt availability. Your server logs show who was limited: which agent, which IP and how many requests.

A 429 says exactly what is going on: too many requests, not a missing page or a missing permission. For Google, that means slowing down instead of dropping URLs, and with Retry-After the server can give a concrete wait time.

The cost is the cost of slower crawling. Google warns that with less crawling it discovers fewer new pages and refreshes existing ones less often, so prices and product availability may take longer to show up in Search. If the 429 lasts for days on the same URLs, they can drop out of the index. And a badly tuned IP limit also blocks people who share a network.

Source: RFC 6585 (April 2012), section 4; RFC 9110 (June 2022), sections 15.5.4 and 15.6.4; Google's status code documentation (accessed 12.11.2026).

Google's sources disagree on how long a deliberate 429 can last. The guide to reducing crawl rate, last updated 6 October 2026, advises against going beyond "1-2 days". Search Console Help on the Crawl Stats report, which shows no date, says not to return 503 or 429 for "more than two or three days". On removal from the index, Google's page on crawling errors, last updated 18 December 2025, says that returning 503 or 429 for more than two days "will cause Google to drop those URLs from the index", while the guide to reducing crawl rate only says the URL may be dropped after multiple days.

The February 2023 Google blog post that made the 429 exception widely known also recommended reducing crawl rate in Search Console. That tool was retired on 8 January 2024, as Google announced in November 2023.

There is also no standard for warning clients before they hit the limit. Headers such as X-RateLimit-Remaining are each service's own convention, and the IETF's RateLimit header fields are still a draft as of November 2026.

• Use your logs to find which agent, IP and URLs are causing the load.
• Before throttling something that claims to be Googlebot, verify it with a reverse DNS lookup or against the IP ranges Google publishes.
• Return 429 only to the excess requests, not to the whole site, and add Retry-After.
• Check which status code your CDN's rate limiting rule returns. In Cloudflare you choose a value between 400 and 499, and the default is 429.
• Keep robots.txt outside the limit.
• Lift the limit on Googlebot after one or two days and check the Crawl Stats report.
• If you are the one receiving the 429, as an API client, wait for the Retry-After time before repeating the request.

• To slow Googlebot down, use 429, 503 or 500, never 403 or 404.
• If the load comes from filter or calendar URLs, fix it in the site structure; a 429 only buys time.

• Rate limiting with 403 in the CDN: Google reads it as "this URL does not exist", not as "slow down".
• Returning 429 on robots.txt.
• Keeping the limit on Googlebot in place for weeks.
• Setting a per-IP limit so low that it blocks users on the same network.

This entry was written in November 2026 with AI assistance. The facts come from RFC 6585 and RFC 9110, Google's documentation on status codes, crawl rate, robots.txt and crawler verification, Search Console Help, Google's blog, MDN, and documentation from Cloudflare, OpenAI, Anthropic, Perplexity and the IETF. It was reviewed before publication and contains no client data.