Crawl budget
Crawl budget is the number of URLs a search engine crawler will visit on a site within a given time frame. It matters primarily on large or complex sites — and worrying about it on small, simple sites is one of the most common technical SEO distractions.
What crawl budget is and when it matters
Crawl budget is determined by two factors:
Crawl demand — how much Googlebot wants to crawl your site based on:
- Perceived value of the content.
- Freshness of page updates.
- Popularity and authority (sites with more inbound links tend to get more crawl attention).
- Number of URLs requesting crawling.
Crawl capacity — how quickly Googlebot can crawl without overwhelming your server. Google is designed to crawl responsibly and will slow down if your server responds slowly or with errors.
When crawl budget matters
Crawl budget is relevant when:
- Your site has millions of URLs.
- New pages are slow to be discovered and indexed.
- Log file analysis shows large volumes of crawler visits to URLs you do not want crawled.
- Your site has significant crawl waste from duplicate URLs, parameters, or thin pages.
Crawl budget does not matter meaningfully when:
- Your site has fewer than 10,000 pages.
- All important pages are indexed and crawled regularly.
- You have no parameter, filter, or session ID URL issues.
Learning objectives
After completing this module, you will be able to:
- Understand when crawl budget is and is not a problem.
- Identify crawl waste patterns and priority crawl paths.
- Improve crawl efficiency on large or complex sites.
Crawl waste: what it is and where it comes from
Crawl waste occurs when Googlebot spends crawl capacity on URLs that have no indexing value. This leaves fewer resources for the pages that matter.
Common sources of crawl waste
URL parameters. Faceted navigation, session IDs, tracking parameters, and sorting parameters can generate thousands of near-identical URLs. Example: a single category page with 50 filter combinations = 50 crawlable URLs with the same content.
Duplicate content. The same content accessible via multiple URLs (www vs non-www, trailing slashes, HTTP vs HTTPS) wastes crawl capacity on content that will not index separately.
Thin or low-quality pages. Pages with little or no useful content get crawled but rarely indexed. Their presence in the crawl queue takes time away from quality pages.
Broken URLs. 404 and 5xx URLs crawled repeatedly are pure waste. They produce errors, not indexed content.
Irrelevant pagination. Deep pagination pages (/page/200) from posts or products from several years ago are often low value and slow to discard.
Infinite crawl spaces. Some calendar systems, site search results, or poorly configured JavaScript applications generate infinite URL chains.
How internal linking affects crawl priority
Googlebot uses internal links to discover and prioritize pages. Pages with many internal links from high-authority pages are crawled more frequently.
To prioritize crawl toward important pages:
- Make key pages accessible from navigation, breadcrumbs, and contextual links.
- Remove or nofollow links to pages you do not want crawled.
- Keep click depth shallow for important pages (within 3–4 clicks from the homepage).
- Fix internal links pointing to 404 or redirected URLs.
Sitemap, robots, canonical, and noindex roles
Sitemaps
XML sitemaps tell Google which URLs exist and which you consider important. Include only canonical, indexable URLs. A sitemap full of redirect URLs, noindex pages, or parameter duplicates sends mixed signals.
Robots.txt
Use robots.txt to block crawling of URL patterns that should never be crawled — not to manage crawl budget for generally useful content. Block:
- Admin and backend paths.
- Staging or development paths.
- Internal search result URLs.
- Session-ID or tracking parameter patterns (when all variants are useless).
Do not block pages you want indexed — Google must crawl a page to index it.
Canonical tags
Canonical tags consolidate crawl and index signals toward preferred URLs. When multiple parameter variants point to the same canonical, Google can learn to prioritize the canonical and reduce wasted crawling of variants.
Noindex
Pages with noindex tell Google not to include them in search results. Googlebot will still crawl them — but over time, crawl frequency on noindexed pages tends to decrease. For large-scale crawl optimization, noindex does not immediately reduce crawl frequency.
Large-site crawl optimization workflow
For sites with crawl budget concerns (typically millions of URLs):
- Analyze log files — identify which URL patterns receive the most bot traffic. Separate useful crawl (priority pages) from waste (parameter variants, thin pages, broken URLs).
- Crawl the site — compare bot behavior in logs with your own crawl to spot discrepancies.
- Identify low-value URL patterns — facets, parameters, filters, archive pages, session IDs.
- Improve internal links — add links to high-priority pages from more authoritative sections.
- Clean sitemaps — remove redirects, noindex pages, and low-value URLs.
- Control parameters — use canonical tags, noindex, robots, or parameter handling where appropriate.
- Monitor bot behavior — re-analyze logs 30–60 days after changes.
Checklist
- Important URLs are in sitemaps (only canonical, indexable URLs).
- Low-value URL patterns (parameters, filters, thin pages) are controlled.
- Internal links from important pages point to other important pages.
- Server errors are monitored and fixed promptly.
- Log data is reviewed for sites with large crawl volume.
Measurement
| Metric | What it tracks |
|---|---|
| Googlebot hits by URL pattern (from logs) | Distribution of crawl activity |
| Crawl frequency of important pages | Whether priority pages are crawled regularly |
| Index ratio vs crawlable URLs | Proportion of crawled pages that get indexed |
| Server error rate (5xx) | Crawl interruptions and reliability issues |
| Time to index new content | How quickly new pages appear in GSC |
Common mistakes
Worrying about crawl budget on small sites. A 5,000-page site with a clean architecture has no meaningful crawl budget problem. Focus effort where it has impact.
Blocking pages with robots.txt before understanding crawl behavior. Blocking too aggressively can prevent Google from reading canonical tags or discovering important pages. Audit what you are blocking and why.
Leaving infinite parameter URLs crawlable. Calendar systems, filter combinations, and sort parameters can create thousands or millions of low-value URLs. Control these with canonical tags, noindex, or parameter rules.
Submitting sitemap with every URL on the site. Sitemaps should be curated. A 2 million URL sitemap that includes thin pages and redirects dilutes the signal about which pages matter.
Confusing crawl budget with crawl rate. Crawl rate is how fast Googlebot fetches pages per second (server capacity). Crawl budget is how many URLs are visited. These are different — server performance improves crawl rate; content quality and link signals improve crawl budget allocation.