A site I worked on shipped a new section in March, and for three weeks none of the new pages showed up in Google. Search Console reported them as “crawled, not indexed.” We had submitted a sitemap, the URLs were reachable, and server logs showed Googlebot hitting them repeatedly. The fix turned out to be nothing obvious. The pages were linked only through a JavaScript-rendered dropdown that ran after hydration, and Googlebot’s second render pass was executing before the menu finished building. The pages were crawled. They were not discovered in the way the crawler expected.

Knowing how search engines crawl your site is the part of SEO that actually belongs to the developer, and it sits underneath everything the content team does. Here is the pipeline, what each stage does, and where small sites get distracted by problems they do not have.

The crawl → render → index pipeline

Three stages, and conflating them is the source of most confusion.

Crawling is fetching. A scheduler picks a URL, a fetcher issues an HTTP GET, and the response body lands in a queue. At this point the search engine has bytes. It does not know what the page is about.

Rendering is executing. If the body contains JavaScript that modifies the DOM, the crawler has to run that JavaScript to see the final HTML. Google uses a workqueue for this. It does not render every page on the same fetch, because rendering is expensive. The page goes onto a queue, a worker eventually loads it in a headless Chromium, runs the scripts, and stores the rendered HTML alongside the original.

Indexing is deciding. The rendered HTML is parsed, extracted into tokens, scored against queries, and either stored in the index or dropped. A page can be crawled and rendered and still not indexed. It can be indexed and still not rank. These are independent gates.

Google documents this in the Search Central crawling and indexing guide. The two-pass detail, where the initial HTML is indexed first and JavaScript waits in a queue, is in their JavaScript SEO guide.

Before a page can be crawled it has to be found. Crawlers discover URLs through three channels.

The first is links. A crawler reads a page it already knows, extracts every <a href> it can see without running JavaScript, and adds new URLs to the crawl queue. Internal link structure matters because it is how the crawler walks your site. Orphan pages, ones with no internal links pointing at them, may never be discovered at all.

The second is sitemaps. A sitemap is an explicit list of URLs you submit, usually an XML file at /sitemap.xml. It is a hint, not a command. The crawler reads it and merges the URLs into its queue, weighting them by how recently they changed and how often you said they update. A sitemap does not guarantee indexing, but it does speed up discovery for new sites and new sections. If you have one, run it through a Sitemap Validator periodically. Sitemaps rot. A reference to a deleted route, a missing <lastmod>, or a malformed namespace will silently make the file less useful.

The third is DNS resolution. When Googlebot fetches a URL it resolves the host to an IP through DNS, and that resolution can fail in ways that look like server errors. A flaky nameserver, a misconfigured CNAME, or a CDN that rate-limits the Googlebot user agent will show up in Search Console as crawl errors that have nothing to do with your HTML.

Crawl budget — and why small sites genuinely don’t need to worry about it

Crawl budget is the number of pages a search engine will crawl on your site in a given timeframe. It is the product of two limits: crawl rate limit, which is how many requests per second Google will hit your server with, and crawl demand, which is how much it wants to crawl based on popularity and staleness.

Google’s own guidance is direct about this. In their crawl budget documentation they say that for sites with fewer than around 10,000 unique URLs, crawl budget is not something to worry about. The crawler will get to everything. If you run a marketing site, a blog, or a small SaaS app, you are almost certainly under that threshold, and reading thinkpieces about crawl budget optimization is a waste of an afternoon.

Crawl budget becomes real for sites in the millions of URLs, or for sites that generate URL variants through faceted navigation, session IDs, or query-parameter permutations. There, the crawler has to choose, and if you feed it ten near-identical URLs for every piece of unique content, you dilute the attention any single page gets. The fix is canonical tags, parameter handling in Search Console, and blocking clearly duplicate paths in robots.txt. None of this applies to a 30-page marketing site.

Rendering and JavaScript-heavy sites

This is the part where modern front-end stacks get into trouble.

Googlebot renders JavaScript, but it does so asynchronously. On the first fetch, it reads the raw HTML response and indexes whatever it finds there. If your page is an empty <div id="root"></div> and everything is hydrated client-side, the first index will see nothing. The page goes onto the render queue, and a worker eventually executes the JavaScript, captures the resulting DOM, and re-indexes.

The lag between fetch and render can be hours. During that window, your page is in the index with whatever the raw HTML contained. On a client-rendered React SPA that is roughly nothing. On an Astro, Next.js static, or server-rendered site, the raw HTML is already complete and the render pass mostly just confirms what was already indexed.

This is why the Astro stack this site is built on matters. Static output means the raw HTML is the final HTML, and Googlebot sees everything on the first fetch. There is no hydration cost to wait out, no render queue to fall behind on, no race between the crawler’s second pass and a client-side menu building itself.

The hydration problem from the opening story is real and common. Links that only exist after a client framework runs, content gated behind user interaction, infinite scroll that loads paragraphs on click. All of these work for humans and partially work for crawlers, and the partial-ness is what bites.

Why crawl rate is not ranking

Googlebot hitting your site every day does not mean you are winning. Crawl frequency reflects how often Google thinks your content changes and how often it wants to refresh its copy of the index. A news homepage gets crawled constantly. A high-ranking evergreen article might be crawled once a month.

People watch their server logs, see Googlebot hitting pages repeatedly, and read it as a positive signal. It is not. It is a mechanical refresh. Conversely, a page that is rarely crawled can still rank first for a competitive query if the content and links support it. Ranking is determined by relevance and authority signals that have almost nothing to do with how often the crawler visits.

If you want to know whether Google is valuing a page, look at impressions and position in Search Console, not at the crawl log. Server logs tell you what the crawler is doing. They do not tell you what the ranker is doing.

Signaling intent with robots.txt

robots.txt is not a security boundary. It is a polite request that well-behaved crawlers honor. It tells crawlers which paths they may fetch and which they should leave alone. Malicious crawlers ignore it. Anything you actually want to protect should be behind authentication, not hidden in a disallow rule.

What robots.txt is good for is steering crawl attention. If you have /admin, /cart, /api, staging paths, faceted-search permutations that explode into thousands of junk URLs, listing them as disallowed keeps the crawler from spending its attention on pages that should never be indexed. The file is small and the syntax is fiddly, so it is worth generating rather than hand-writing. Use a robots.txt generator and read the resulting file before you publish it. A misplaced wildcard will block the entire site, and Google will usually obey the request within hours.

Pair robots.txt with the sitemap. The robots file has a Sitemap: directive you can use to advertise your sitemap location to any crawler that reads the file. It is the canonical place to announce the sitemap alongside the per-path rules.

FAQ

Does Google crawl JavaScript?

Yes. Googlebot executes JavaScript using a headless Chromium instance, but it does so in a second pass after the initial HTML is fetched. The render queue introduces a delay, sometimes hours. Server-rendered or statically generated HTML gets indexed on the first fetch, which is why statically generated sites tend to be more reliably indexed.

Probably, for new content. Internal links let the crawler discover pages by walking, but a sitemap short-circuits that walk and tells the crawler about new or updated URLs directly. For sites with frequent new pages, the sitemap is how those pages get discovered quickly. For a site that almost never changes, internal links alone are usually enough.

Will robots.txt keep my staging site out of Google?

Not reliably. Well-behaved crawlers honor robots.txt, but anyone can ignore it, and if any external link points at the staging host Google may index it anyway. The safe approach is authentication on the staging environment plus a noindex HTTP header, not a Disallow: / in robots.txt.

How often will Google recrawl my pages?

It depends on crawl demand, which is driven by how often your content changes and how popular the site is. A news site may see crawls every few minutes. A stable marketing page might be revisited every few weeks. You cannot directly request a higher crawl rate beyond submitting fresh content and updating <lastmod> honestly in the sitemap.

The question that lingers

We know the pipeline. We know the stages and where they fail. What nobody outside the search engines can tell you is how much of the index is decided by rendering fidelity versus by link authority versus by something the ranker does that has never been published. You can build a perfectly crawlable, perfectly rendered, perfectly linked site, submit an impeccable sitemap, and still lose to a competitor whose HTML is worse but whose backlink profile is older. The crawl side is understood. The ranking side is not. So the real question, the one nobody has a clean answer for, is how much of what we call technical SEO actually moves the needle versus just removes self-inflicted wounds.