Crawl budget

Do you need to care?

Google says sites without many rapidly changing pages, or whose pages are crawled the day they are published, don’t need its guide. For Google Search, “keeping your sitemap up to date and checking the Page Indexing report regularly is adequate.”

Officially documented · Google

The guide is intended primarily for three kinds of site. The first is large sites (1 million+ unique pages) with content that changes moderately often (once a week). The second is medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily). The third is sites with a large share of URLs reported as “Discovered – currently not indexed”.

Officially documented · Google

What determines it

“A site’s crawl budget is determined by two main elements: crawl capacity limit and crawl demand.” The capacity limit, also called hostload, caps how much time the server spends on Google’s connections so crawling doesn’t overload it. Every site starts with the same conservative default. Separate hostnames such as www.example.com and code.example.com have separate crawl budgets.

Officially documented · Google

Crawled is not indexed

“Not every page that is crawled will necessarily be indexed.” After crawling, each page is evaluated, consolidated and assessed for suitability for the index.

Officially documented · Google

Related

• HTTP status codes and crawlers — 5xx and 429 responses slow crawling down.
• XML sitemaps — the minimum Google suggests for everyone.