HTTP status codes and crawlers

Summary

Code	Google’s crawlers
2xx	Content considered for indexing — not guaranteed
301	Followed; strong signal the target should be processed
302	Followed; weak signal
4xx (not 429)	Not indexed; previously indexed URLs removed; no effect on crawl rate
429, 5xx	Crawling slows; indexed URLs kept for a while, eventually dropped

2xx

“For Google Search, an HTTP 2xx (success) status code doesn’t guarantee indexing.” Some 2xx pages look like errors, such as an empty page or an error message. Search Console reports those as soft 404s.

Officially documented · Google

3xx

For a 301, Google follows the redirect and uses it “as a strong signal that the redirect target should be processed”. For a 302, Google follows it and uses it as a weak signal.

Officially documented · Google

4xx

Google doesn’t index URLs that return a 4xx status code. Already indexed URLs that start returning one are removed. All 4xx errors except 429 are treated the same and have no effect on crawl rate. So Google advises: “don’t use 401 and 403 status codes for limiting the crawl rate.”

Officially documented · Google

429 and 5xx

“5xx and 429 server errors prompt Google’s crawlers to temporarily slow down with crawling.” Already indexed URLs stay in the index for a while but are eventually dropped. Once the server returns 2xx again, Google gradually increases the crawl rate.

Officially documented · Google

For AI search

Google’s AI features draw on the same index (How Google Search works). A page that fails with 4xx or prolonged 5xx drops out of AI Overviews along with classic results. Bot-protection layers that answer crawlers with 403 or 429 have the same effects as these errors. See AI search access audit.