Skip to main content

Crawling vs indexing

Crawling and indexing are two distinct steps in the process that takes a URL from existing on the web to appearing in search results. Confusing them leads to incorrect diagnoses and the wrong fixes. This module explains the full pipeline — from discovery to ranking — and how to diagnose failures at each stage.

The search engine pipeline: four stages​

Understanding the pipeline is the foundation of all technical SEO diagnosis.

1. Discovery​

Search engines discover URLs through:

  • Links from already-crawled pages (internal and external).
  • XML sitemaps submitted through Search Console.
  • Direct URL submission via the URL Inspection tool.

If a URL is not discovered, it cannot be crawled. Orphan pages with no internal or external links and no sitemap inclusion are invisible to crawlers.

2. Crawling​

Crawling is the process of Googlebot visiting a URL, downloading the page's HTML, and following links. A page being crawled does not mean it will be indexed.

Crawlability can be blocked by:

  • robots.txt disallow rules.
  • Redirect loops or broken redirect chains.
  • Login walls or access controls.
  • Server errors (5xx).
  • Crawl budget exhaustion on large sites.

3. Rendering​

After crawling, Google renders the page — executing JavaScript to see the final visible content. This is a separate processing step from downloading HTML. Pages that rely heavily on JavaScript for content may be crawled but not fully rendered until a second pass.

4. Indexing​

Indexing is Google's decision to include a URL in its searchable index. A page can be crawled but not indexed for many reasons. Indexing can be blocked by:

  • noindex meta tag or HTTP header.
  • Canonical tag pointing to a different URL.
  • Page being assessed as low quality or duplicate.
  • Being in the "crawled, currently not indexed" state (quality signal).

5. Ranking​

Only indexed pages can rank. Ranking is a separate process that evaluates relevance, authority, and quality signals against user queries. A page can be indexed without ranking well — this is a content and authority problem, not a crawling or indexing problem.

Learning objectives​

After completing this module, you will be able to:

  • Explain the difference between crawling, rendering, indexing, and ranking.
  • Diagnose why important URLs are not indexed.
  • Choose the correct fix for discovery, crawl, render, and index issues.

Crawlable vs indexable pages​

A page can be crawlable (accessible to bots) but not indexable (blocked from the index). A page can be indexable (no noindex) but not actually indexed (quality or canonicalization decision). These are distinct states.

StateCan be crawledCan be indexed
Blocked by robots.txtNoNo
noindex tagYes (unless also blocked)No
Canonical points elsewhereYesTypically not (deferred to canonical)
Login wallNoNo
Live, no issuesYesYes (if quality is sufficient)

Common URL-level blockers​

robots.txt​

The robots.txt file can disallow crawling of specific paths. If Googlebot is blocked from a URL, it cannot crawl or index it — even if the URL is in a sitemap. Common mistake: blocking important pages in robots.txt while also listing them in sitemaps.

noindex​

The noindex directive in the HTML <head> or HTTP response header tells Google not to index the page. The page can still be crawled (unless also blocked by robots.txt). Important: Google must be able to crawl the page to read the noindex directive.

Canonical tags​

A canonical tag <link rel="canonical"> signals the preferred version of a page. Google typically indexes the canonical URL and defers the others. If a canonical tag points to a different URL, the current page will likely not appear in search results.

Redirects​

Pages that redirect (301, 302) are not directly indexed — the destination URL is. A chain of multiple redirects before reaching the final destination adds delays and can cause issues. Redirect loops create permanent crawling failures.

Login walls​

Content behind authentication (login, paywall) cannot be crawled. Subscription content should use the appropriate meta tags to signal this to Googlebot where applicable.

URL Inspection workflow​

Google Search Console's URL Inspection tool is the primary diagnostic for crawl and index issues.

Step-by-step:​

  1. Enter the full URL in the URL Inspection search bar.
  2. Check the indexing status — is it indexed? If not, what reason is given?
  3. Check Discovery — how did Google find the URL (sitemap, referred by)?
  4. Check Crawl — when was it last crawled? What User-Agent was used?
  5. Check Enhancements — any schema or structured data issues?
  6. Click Test live URL to see what Google renders at that moment — compare to the actual page.
  7. Review More information for any notes on why it was not indexed.

Diagnosing common states​

Discovered — currently not indexed​

The URL has been found but Google has decided not to crawl or index it yet. Common causes:

  • Crawl budget is being prioritized elsewhere.
  • Page is linked from low-priority pages.
  • Site has many pages with similar or thin content.

Actions: Improve internal linking from higher-authority pages, add the URL to a sitemap, reduce thin content on the site.

Crawled — currently not indexed​

Google crawled the page but chose not to index it. This is often a quality signal. The page may have:

  • Thin, duplicate, or low-value content.
  • Content that does not meet helpful content expectations.
  • Issues with page quality signals.

Actions: Substantially improve content quality, differentiate from similar pages, add authoritative internal links.

Indexed — not submitted in sitemap​

Google indexed the page despite it not appearing in a sitemap. Review whether this page should be canonical and whether it should be in your sitemap.

Checklist​

  • URL is discoverable through links or sitemaps.
  • URL is crawlable (not blocked by robots.txt or technical errors).
  • URL is indexable (no noindex, appropriate canonical).
  • Canonical target is correct and intentional.
  • Rendered content is visible and matches the page intent.

Measurement​

MetricWhat it tracks
Indexed URL count (GSC Coverage)Total pages in Google's index
Sitemap indexed/submitted ratioSitemap quality and index acceptance
Crawl status in GSCDistribution of crawl and index states
URL Inspection resultsPage-level diagnosis
Search appearancePresence in SERP and rich results

Common mistakes​

Assuming crawled means indexed. Googlebot can visit a page and decide not to index it. The two states are independent.

Blocking URLs with robots.txt before Google can read canonical/noindex tags. If Googlebot cannot access the page, it cannot read the noindex directive or canonical tag. The correct approach is to allow crawling of pages you do not want indexed but use noindex — not to block them in robots.txt.

Ignoring rendered HTML. JavaScript-rendered content may not appear in the crawled version of a page if rendering is delayed. Always check the rendered version in URL Inspection.

Listing noindex pages in XML sitemaps. Sitemaps should only include pages you want indexed. Submitting noindex pages in a sitemap sends a contradictory signal.

Confusing ranking issues with indexing issues. A page that is indexed but not appearing in results for specific queries has a ranking problem — addressed through content quality, relevance, and authority — not a crawl or index problem.