Robots.txt Generator

What robots.txt actually is — and is not

Robots.txt is a text file at the root of your site (https://example.com/robots.txt) that gives advice to well-behaved crawlers. It is defined by RFC 9309, published in September 2022, which formalized the de-facto protocol Martijn Koster introduced in 1994.

The critical thing to understand: robots.txt is not a security control. It is a cooperative opt-out. It tells Googlebot, Bingbot, and other compliant crawlers which paths to skip — but it cannot stop a scraper that chooses to ignore it. Anything you want truly hidden must sit behind authentication or access control, not in robots.txt.

• RFC 9309 defines only User-agent, Allow, Disallow, and Sitemap — nothing else is standardized
• The file is fetched once per crawl session and cached; changes take hours to days to propagate
• A 5xx (server error) response makes Googlebot assume the entire site is off-limits and pause crawling

The wildcard and end-of-path anchor

Two pattern characters extend matching beyond exact paths. * matches any sequence of characters (like a glob), and $ anchors a rule to the end of the URL. Together they let you target URL patterns precisely.

Precedence is most specific wins: a longer match beats a shorter one, and Allow beats Disallow when both match a path equally. This is how you carve out exceptions inside a blocked tree.

User-agent: *
Disallow: /search*          # blocks /search, /search?q=x, /search/advanced
Disallow: /*.pdf$           # blocks any URL ending in .pdf
Allow:    /search/about$    # but allow exactly /search/about

Why crawl-delay is dead — and what to use instead

The Crawl-delay directive was never part of any formal spec. RFC 9309 deliberately omits it because implementations never agreed on what it meant — some treated it as seconds between requests, others as requests per second. Google has never honored it.

If Googlebot is hammering your server, the correct lever is the Crawl Rate setting in Google Search Console (Settings → Crawl stats), not a directive in robots.txt. Bing offers a similar control in Bing Webmaster Tools.

robots.txt Disallow versus the noindex directive

These are commonly confused but do opposite things. Disallow in robots.txt tells crawlers not to fetch a URL — but if other pages link to it, the URL can still appear in results as a bare "uncrawled" entry. The noindex meta tag or HTTP header tells crawlers they may fetch the page but must not index it, which fully removes it from results.

Rule of thumb: if a page must never appear in search, let crawlers fetch it and use noindex. If you need to preserve crawl budget or block access entirely, use Disallow. Never combine them — Disallow hides the page, which means Googlebot never sees your noindex tag and cannot honor it.