What Is Crawl Budget?

Crawl budget is the number of pages a search engine will crawl on your website in a given period. Broken links, 404 errors, and redirect chains waste it — leaving your important pages under-crawled.

Glossary·Updated June 2, 2026

Definition: Crawl budget is the finite number of URLs that a search engine’s bot (like Googlebot) will crawl on your website within a given timeframe. Search engines allocate crawl capacity based on your site’s size, speed, authority, and error rate. Efficient sites get their important pages crawled more frequently.

How crawl budget is determined

Google’s Googlebot decides how often and how deeply to crawl your site based on two factors:

  • Crawl rate limit — how fast Googlebot can crawl without overloading your server. Faster servers get crawled more aggressively.
  • Crawl demand — how often Google thinks it needs to re-crawl your pages based on their freshness, popularity, and whether they have changed.

Together, these create your effective crawl budget: the volume of pages Google will process from your site in a given period.

How broken links waste crawl budget

Every time Googlebot requests a URL and receives a 404 error, it uses one crawl request without indexing anything useful. For a site with 100 broken internal links, Googlebot might spend 10–20% of its crawl budget on dead pages. Those are requests it is not spending on your new content, updated articles, or important product pages.

Fixing broken links with 301 redirects eliminates these wasted requests. Read the full guide on how to fix broken links in WordPress.

Other things that drain crawl budget

  • Redirect chains — A → B → C uses multiple crawl requests to follow one path.
  • Duplicate content — Multiple URLs serving identical content (e.g. with and without trailing slash, HTTP vs HTTPS).
  • Thin or low-value pages — Pages with little content that Google does not need to index.
  • Infinite faceted navigation — E-commerce filters that generate thousands of unique URLs.

How to improve your crawl budget efficiency

  1. Fix all broken links and 404 errors (use FixLinks AI to find and redirect them).
  2. Consolidate redirect chains so each old URL points directly to its final destination.
  3. Add noindex to thin, duplicate, or utility pages you do not want indexed.
  4. Improve page speed — faster sites get crawled more efficiently.
  5. Keep your XML sitemap accurate and free of 404 or noindex URLs.

Frequently asked questions

Does crawl budget matter for small websites?

For small sites (under a few hundred pages), crawl budget is rarely a significant concern — Google crawls small sites fully and regularly. It becomes important at scale: large e-commerce stores, news sites with thousands of articles, and sites with lots of duplicate or thin content.

How do I increase my crawl budget?

Eliminate 404 errors and fix broken links (they waste crawl requests), remove or noindex thin/duplicate pages, fix redirect chains, and improve site speed. Faster, error-free sites are crawled more efficiently.

Do broken links waste crawl budget?

Yes. Every 404 error uses a crawl request without contributing anything useful. On a site with hundreds of broken links, a meaningful portion of your crawl budget is being consumed by dead URLs instead of new or updated content.