noindex vs robots.txt vs canonical: when to use which

Three controls, robots.txt and noindex and canonical, get conflated constantly. They appear in the same SEO audit reports, they all sound like “tell Google what to do with my pages,” and they are not interchangeable. The noindex vs robots.txt confusion in particular wastes real ranking signal: I’ve seen site owners block a staging path in robots.txt, expect those URLs to drop out of the index, and then watch them sit there for months as URL-only listings. This article gives you the decision rule for which control handles which job, with the interaction traps that bite when you mix them wrong.

robots.txt prevents crawling (and why that is different from deindexing)

robots.txt is the crawler’s gate. Defined by RFC 9309, it tells fetching user-agents which paths they may not request. If you disallow a path, Googlebot does not fetch it. That is the entire contract.

What it does not do is remove the URL from the index. Google can still list a URL it has never crawled, using whatever fragmentary signal it has: an inbound link, a reference in a sitemap, anchor text from another page. You’ve seen these in search results: a bare URL, no title, no snippet, with the note “A description for this result is not available because of this site’s robots.txt.” That is the URL-only listing, and it ranks for almost nothing but it is technically indexed.

A robots.txt disallow looks like this:

User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /cart/public

Crawl budget is the legitimate reason to use it at scale: large e-commerce sites, faceted navigation gone wild, parameter explosions. For a site with a few hundred URLs, you do not need a clever robots.txt for crawl-budget reasons. You can generate one with the Robots.txt Generator and validate the final file against your actual paths before you ship it.

noindex prevents indexing but allows crawling

noindex is the indexer’s gate. The crawler still fetches the page; the indexer reads the directive and drops the URL from results. Two ways to send it, both documented in Google’s crawlers and indexing guidelines:

<meta name="robots" content="noindex">

in the document head, or as an HTTP header:

HTTP/1.1 200 OK
X-Robots-Tag: noindex
Content-Type: text/html

The header form is what you want for PDFs, images, and anything where you cannot easily edit HTML. It also wins on consistency: a single server-level rule covers every response regardless of how the page was rendered.

The key property: the page must be crawlable for noindex to work. If robots.txt blocks the crawler, Googlebot never fetches the page, never sees the noindex directive, and falls back to whatever signal it has, which is usually a URL-only listing. This is the trap that produces the next section.

You can verify a noindex directive the same way you’d verify any meta tag on a live page: fetch the URL with the Meta Tag Checker and confirm the directive is actually present in the rendered head. A noindex injected by JavaScript five seconds after load has historically been unreliable; put it in the initial HTML.

canonical picks the canonical URL among a duplicate cluster

canonical does not keep a page out of the index at all. It tells the indexer which URL is the authoritative version among a set of near-duplicates, so the cluster consolidates ranking signals onto one URL instead of fragmenting them.

<link rel="canonical" href="https://example.com/dresses/blue">

It is a hint, not a command. Google can and does ignore it when the signals disagree. If every page in a cluster points its canonical at itself, or if the canonical target is itself noindexed, Google falls back to its own duplicate detection and picks what it thinks is canonical. Bing has historically treated canonical more loosely still.

The correct use is dedupe: print-friendly versions, sorted/filtered variants of a listing, http/https and www/non-www pairs before redirects are in place, regional variants of the same content. If you have one product page reachable through five tracking-parameter URLs, canonical collapses them.

It is not the right tool for “I don’t want this page indexed.” That’s noindex. And it is not the right tool for “I don’t want this page crawled.” That’s robots.txt.

The interaction trap

Here is the single most-misunderstood combination in technical SEO.

A site owner reads about robots.txt, sees Disallow: /staging/, and applies it. They also want staging out of the index, so they add <meta name="robots" content="noindex"> to the staging template. They check Search Console two weeks later and the staging URLs are still indexed as bare URLs, no title, no snippet. They conclude noindex doesn’t work.

What happened: robots.txt blocked the crawler. Googlebot honored the disallow, never fetched the page, never read the noindex meta tag. Without being able to crawl the page, Google fell back to its other signals (the staging subdomain was linked from a public issue tracker, or a sitemap got published accidentally) and kept the URL in the index as a URL-only listing.

The fix is to allow crawling of URLs you want deindexed, and use noindex as the gate. Googlebot fetches the page, reads noindex, drops the URL. Once the URL is gone from the index, verified in Search Console’s URL Inspection tool, you can re-add the robots.txt disallow if crawl budget matters. The sequence is noindex first, then (optionally) block crawling.

There is a related footgun: noindex, follow. The follow directive is meaningful. It tells the crawler to still use the links on the page for discovery and PageRank flow even though the page itself won’t be indexed. This is what you want for paginated archives and filtered navigation pages where the listing page itself shouldn’t rank but the products it links to should. noindex alone, without follow, leaves link behavior to default, which is fine — the default is index, follow and it has been stable for many years, so the follow behavior is what you already get. Being explicit with noindex, follow is a style choice, not a hedge against Google changing the default.

A decision table

Goal	Use	Notes
Remove a page from Google’s index	noindex (meta tag or X-Robots-Tag)	Must allow crawling. Confirm via URL Inspection.
Stop Googlebot from fetching a path	robots.txt Disallow	URL may still appear as a URL-only listing.
Collapse duplicate variants onto one URL	canonical	Hint, not a command. Use for sorting/filter/print variants.
Block staging/dev sites from results	noindex (with crawl allowed)	Common mistake: block crawl first, URLs stay indexed.
Block a script or asset from crawling	robots.txt Disallow	But don’t block JS/CSS that affects rendering.
Suppress index of a PDF or image	X-Robots-Tag: noindex header	Only the header form works for non-HTML.
Tell Google a page has permanently moved	301 redirect	Not a job for canonical. canonical is for near-duplicates, not relocations.
Block a page permanently and remove from index	both, in order: noindex first, then Disallow after deindexed	Sequence matters.
Paginated archive pages that shouldn’t rank but link to products	noindex, follow	Page drops from index; links still pass signal.
Parameterized URL explosion (?sort=, ?sessionid=)	canonical to the clean URL, or noindex on the variants	Or Google’s URL Parameters tool historically, now removed.

The pattern across the table: robots.txt answers “may I fetch,” noindex answers “may I index what I fetched,” canonical answers “of these duplicates, which one counts.” Different questions, different tools.

Wire-up

For the crawl side of this, generating and checking the robots.txt itself, use the Robots.txt Generator. It produces syntactically correct rules with User-agent, Disallow, and Allow lines, and is faster than hand-editing when you’ve added a new section of the site.

To confirm what a live page is actually sending to crawlers, meta tags, X-Robots-Tag header, canonical link, run the URL through the Meta Tag Checker. This is the diagnostic step people skip and then spend a week wondering why a page isn’t behaving.

If you’re using a sitemap to push URLs at Google (which is the right move when you want fast discovery of new or updated pages, and which has no conflict with any of these directives as long as the sitemap only lists canonical, indexable URLs), validate it with the Sitemap Validator before submission.

For the deeper crawl-vs-index mechanics, including what happens when you disallow a URL that’s already been indexed and what the timeline looks like, I wrote a separate piece on the robots.txt vs noindex distinction that focuses specifically on that pairing.

FAQ

If I disallow a URL in robots.txt, will Google remove it from the index?

No. robots.txt blocks crawling, not indexing. Google may keep the URL in the index as a URL-only listing if it has any signal at all for the URL. To actually remove a URL, use noindex and allow the crawler to fetch the page so it can read the directive.

Can I use canonical to keep a page out of the index?

No. canonical picks one URL out of a duplicate cluster and consolidates ranking signal onto it; it does not deindex anything. If the goal is “don’t index this page,” use noindex. canonical is for dedupe, not suppression.

What’s the right way to block a staging site?

Put noindex on every staging page via meta tag or X-Robots-Tag header, and let Googlebot crawl those pages. Do not block staging with robots.txt first; that prevents Google from reading the noindex and leaves you with URL-only listings. Optionally add HTTP basic auth on top, which is a harder block than any robots directive.

Does noindex work on PDFs and images?

Yes, but only via the X-Robots-Tag: noindex HTTP header, not a meta tag. Meta tags live in HTML, and PDFs and images don’t have an HTML head. Configure the header on the server (or CDN) for any non-HTML response you want suppressed from the index.