If you want a page to disappear from Google, your first instinct is probably to block it in robots.txt. That instinct is wrong — or at least incomplete — and it is the single most common SEO mistake made on staging sites, thin content, and pages that should never have been indexed. The two mechanisms do fundamentally different things, and combining them produces the opposite of what you want.

The core distinction

robots.txt controls crawling. The noindex directive controls indexing. These are not the same stage of the pipeline.

When you add a path to robots.txt:

User-agent: *
Disallow: /admin/
Disallow: /staging/

you are telling Google’s crawler, “do not fetch this URL.” Googlebot will honour that and skip the request. But here is the catch: Google can still index a URL it has never crawled. If other sites link to /staging/some-page, Google may list that URL in search results as an uncrawled entry — a bare link with no title or snippet, derived purely from the anchor text of inbound links. You blocked crawling and got a result in the index anyway.

When you instead serve a noindex directive — either as a <meta> tag or an X-Robots-Tag HTTP header — you let Google crawl the page, read the directive, and drop it from the index entirely. The page vanishes from search results even though Googlebot is allowed to visit it.

Why mixing them is the worst of both worlds

This is the trap. Suppose you both disallow the path in robots.txt and put noindex on the page. What happens?

Google respects robots.txt first. It never fetches the page, so it never sees the noindex directive. The directive cannot take effect because the crawler never reaches it. You have disabled the very mechanism you were relying on. The page stays in the index as an uncrawled URL, exactly as if you had used robots.txt alone.

The rule, stated plainly: if you want a page removed from the index, it must be crawlable. Let Googlebot in, serve noindex, and let the directive do its job. Reserve robots.txt Disallow for pages you genuinely never want fetched — internal admin panels, search-result pages that generate infinite crawl loops, or endpoints that waste crawl budget.

The two ways to send noindex

Inside HTML, in the <head>:

<meta name="robots" content="noindex">

Or as an HTTP response header, useful for non-HTML resources like PDFs:

X-Robots-Tag: noindex

Both are equivalent. The header form is the only option for binary files and API responses where you cannot inject a <meta> tag.

You can combine directives: noindex, follow drops the page from the index but still lets Google follow its outbound links and pass link equity onward — often the right choice for thin pages that link to valuable ones.

Robots.txt is advisory, not security

This bears repeating: robots.txt is a request, not a gate. Well-behaved crawlers (Google, Bing, most legitimate search engines) honour it. Malicious scrapers, archiving tools, and anyone writing curl ignore it completely. Never use robots.txt to hide sensitive content. A Disallow: /private/ line is effectively a roadmap telling bad actors exactly where the interesting stuff is.

Real access control belongs behind authentication and authorisation — not in a publicly readable text file. If a URL must not be seen, it must require login at the server level.

Crawl-delay is dead

Many older tutorials recommend a Crawl-delay directive in robots.txt to slow Google down. RFC 9309, published in September 2022 as the first formal IETF standard for the Robots Exclusion Protocol, does not include crawl-delay. Google has ignored it for years; Bing still reads it but is increasingly the exception. The directive is legacy, unsupported by the standard, and should not be relied on. If you need to throttle crawl rate, use Google Search Console’s crawl-rate setting instead.

RFC 9309 also formalised what was already true: noindex as a robots.txt directive was never part of the standard, and Google dropped support for it in 2019. Controlling indexing from inside robots.txt simply does not work — use the meta tag or HTTP header.

The Sitemap directive

One directive worth adding to robots.txt is the Sitemap line, which tells crawlers where your XML sitemap lives:

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap-index.xml

This is the canonical way to advertise your sitemap. It complements the noindex strategy cleanly: use noindex to remove unwanted pages from the index, and a sitemap to make sure the pages you do want are discovered promptly. You can build a correct robots.txt with the robots.txt generator and confirm your sitemap is valid and internally consistent with the sitemap validator before publishing.

The right pattern for staging and thin pages

For most “hide this from Google” scenarios, the answer is straightforward:

  1. Keep the page crawlable — do not disallow it in robots.txt.
  2. Serve noindex via <meta> tag or X-Robots-Tag header.
  3. Remove any existing robots.txt block on that path so the crawler can actually read the directive.
  4. Optionally add the URL to the sitemap with noindex in place — Google will crawl it faster and process the drop sooner.

For pages you never want fetched at all — admin, internal search, duplicate parameter URLs — Disallow in robots.txt is correct, and you accept that an uncrawled link could still appear if externally linked. For everything else where removal from the index is the goal, noindex alone is the answer.

Understand the pipeline — crawl, then index — and the right tool becomes obvious. Misunderstand it, and you will spend months wondering why a page you “blocked” is still showing up in search results.