A robots.txt file is the first thing a crawler fetches on your site, and the one it consults before requesting anything else. This robots.txt guide covers what the file actually does, every directive in the spec, how pattern matching works, and the mistakes I keep seeing devs ship to production. The format was codified in 1994 as an informal standard and finally formalized in RFC 9309 in September 2022, so the rules are now written down rather than passed down as folklore.
What robots.txt actually does
Robots.txt is a request, not a fence. The crawler reads it, decides which URLs it will not fetch, and moves on. A crawler that wants to ignore it can, and malicious scrapers do. Do not put anything sensitive behind a Disallow line and consider the door closed. Authentication, authorization, and rate limiting are the controls that actually protect data; robots.txt is how you talk politely to crawlers that listen.
For crawlers that do listen, the file controls fetching, not indexing. That distinction matters more than any other in this article. A Disallow tells Googlebot not to crawl a URL, but if that URL is reachable from elsewhere and has clear signals pointing at it, Google can still index it on the strength of those signals. To keep a page out of the index entirely you need a noindex rule or response header, not a Disallow. We unpack that exact confusion in our robots.txt vs noindex write-up.
The file lives at /robots.txt on the root of the host, served as text/plain, with a 200 status. If the file does not exist the crawler treats that as “no rules.” A 4xx other than 404 is treated as “no rules” too. A 5xx is where the spec and real crawlers part ways: RFC 9309 §2.3.1.4 says a 5xx leaves the robots.txt undefined and the crawler MUST assume complete disallow — the opposite of “no rules.” Googlebot’s practical behavior is friendlier: it pauses crawling for roughly the first 12 hours, and if the 5xx persists it then treats the file as “full allow.” Either way, serving robots.txt off a flaky backend is a self-inflicted wound.
The directives
Five directives do real work. RFC 9309 defines three of them (User-agent, Allow, Disallow); the other two (Sitemap, Crawl-delay) are de facto standards that every major crawler parses.
User-agent starts a group. The value is matched against the crawler’s own user-agent token, case-insensitively and by substring on tokens. User-agent: * is the wildcard group, applied to any crawler that has no more specific group. A group applies to a crawler only if that crawler has at least one rule it can match, so order matters less than specificity.
Allow and Disallow are the meat of the file. Both take a path value. Disallow: /private blocks crawling of any URL whose path begins with /private. An empty Disallow: means “allow everything” and is the explicit way to say a group has no restrictions. Within a single group, the most specific match wins, so Allow: /private/public defeats Disallow: /private for that one path.
Sitemap is a top-level directive, not part of any group. You can list as many as you want, with absolute URLs.
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-images.xmlCrawl-delay sets a minimum number of seconds between requests from a single crawler. Bing and Yandex honor it. Google ignores Crawl-delay entirely. It never officially supported the directive, and in 2019 its open-source robots.txt parser stopped parsing it at all. Including it costs nothing, but it is not a throttle you can rely on for Google traffic.
Pattern matching
The path value in Allow and Disallow supports two wildcards. An asterisk * matches any sequence of characters. A dollar sign $ anchors the pattern to the end of the path. Without $, rules are path-prefix matches, so Disallow: /a blocks /a, /about, /admin, and /apple-pie.
Concrete examples do more than prose here.
User-agent: *
# Block any URL containing "admin"
Disallow: */admin*
# Block only the exact path /search
Disallow: /search$
# Block URLs ending in .pdf
Disallow: /*.pdf$
# Allow search-result pages but block faceted filters
Allow: /search$
Disallow: /search?A pattern that begins with * is the way to express “anywhere in the path.” A pattern that ends with $ is the way to express “exactly this path.” Everything else is a prefix. Get those three cases right and you have covered 90 percent of what people want robots.txt to do.
Two subtleties catch people. First, the path you match against is the URL’s path and query, with percent-encoding normalized. Second, RFC 9309 says a wildcard matches zero or more characters, so Disallow: /a*b blocks /ab and /a-very-long-thing-b alike. There is no concept of “exactly one segment” in the spec. If you need that, you list the segments explicitly or handle it with a noindex rule on the pages themselves.
Common mistakes that block indexing
I keep seeing the same three patterns in robots.txt files that ship to production and quietly sink a site’s search presence.
Disallow: / blocks the entire site. This is the correct line for a staging environment. It is the worst possible line for a production site, and it gets there most often because someone copy-pasted a staging config, or flipped an environment flag the wrong way, and Google takes the file at its word. The fix is one character: Disallow: with nothing after it, or the line deleted entirely.
The second mistake is blocking JS and CSS. Modern Googlebot renders pages to understand them, and rendering requires executing the JavaScript and applying the stylesheets. If robots.txt disallows /assets/, /static/, or /wp-content/, Googlebot will render a broken page and may treat your content as low quality. The rule of thumb is: allow everything a real browser needs to render above-the-fold content.
The third mistake is trailing-slash confusion. Disallow: /blog blocks both /blog and /blog/post, because it is a prefix match. Disallow: /blog/ blocks /blog/ and everything under it but leaves /blog itself crawlable. If your canonical URLs are slashless, the /blog form is what you want when the intent is “block the blog.” If your canonicals are slashful, you need /blog/. Pick the one that matches your routing.
A mistake I see less often but worth mentioning: case sensitivity. Paths in robots.txt are case-sensitive. Disallow: /Admin does not block /admin. Most servers are case-insensitive on Windows and case-sensitive on Linux, and most crawlers run on Linux, so assume case matters.
How crawlers actually read it
Googlebot fetches /robots.txt once per 24 hours and caches the result. Changes you make take up to a day to propagate, sometimes longer if the previous fetch set a long cache header, and there is no “submit” button for robots.txt the way there is for sitemaps. If you need a faster refresh, the last-resort move is a temporary rename and a 404, which Google treats as “no rules” within hours. Use that only when you understand the consequence, which is that everything becomes crawlable.
When Googlebot has a fresh copy, it consults it before every crawl request. For each URL it picks the most specific matching rule, with Allow winning ties over Disallow when both match. Rules from different groups are never merged; a crawler uses exactly one group, chosen by its user-agent token, falling back to * if nothing specific matches.
The fetch-and-render pipeline then works in two phases. Phase one is the crawl fetch, governed by robots.txt. Phase two is rendering, where Googlebot executes JavaScript, applies CSS, and produces a picture of the page closer to what a user sees. Robots.txt gates phase one. Rendering can be gated by noindex, but not by anything in robots.txt beyond the asset rules mentioned above.
For a crawler other than Google the picture differs in detail but not in shape. Bing honors Crawl-delay. Yandex has stricter pattern rules. Baidu is less transparent. The throughline is: write to the RFC 9309 spec, test against Googlebot, and accept that smaller crawlers will interpret the same file slightly differently.
A useful diagnostic when traffic from a specific crawler drops to zero: check whether the file declares a group for that user-agent and whether that group contains a Disallow that matches the site root. I once watched a client lose all Bing traffic for a week because someone had added a User-agent: Bingbot group with Disallow: / while testing a redirect. Bingbot obeyed the rule; the client never noticed because Google still worked. Search Console’s URL Inspection tool will tell you, for Googlebot specifically, whether a given URL is blocked by robots.txt, and that is the fastest way to confirm a hypothesis.
A worked example for a dev-tools site
This is the configuration I would ship for a static developer-tools site like this one. The intent is to allow everything by default, block a handful of noisy paths, and publish the sitemap.
# robots.txt for a dev-tools site
User-agent: *
Allow: /
# Internal search results, keep out of the index
Disallow: /search
Disallow: /search?
# Faceted filter paths that produce infinite URL space
Disallow: /*?sort=
Disallow: /*?filter=
# Generated preview endpoints used by client-side tools
Disallow: /preview/
# Allow the assets that rendering needs
Allow: /assets/
Allow: /static/
# Sitemaps
Sitemap: https://devtidy.com/sitemap-index.xml
Sitemap: https://devtidy.com/sitemap-blog.xmlThree notes on this file. The Allow: / at the top is belt-and-braces; combined with User-agent: * it makes the default explicit, which helps the next person reading the file. The Disallow: /search line protects against infinite-index problems caused by facet URLs. The Allow: /assets/ line is not strictly necessary because nothing above it blocks that path, but documenting intent here saves a future maintainer from accidentally breaking rendering when they add a broader Disallow.
If you want to generate a file like this for your own site without writing it by hand, the Robots.txt Generator takes your sitemap URL, the paths you want to block, and the user-agents you want to target, and emits a ready-to-deploy file. Pair it with the Sitemap Validator to make sure the sitemap referenced from robots.txt is itself well-formed, and with the Meta Tag Checker to confirm the pages you allow through are emitting the indexing directives you intend. The last pairing matters because robots.txt and noindex are two halves of one decision: which pages should appear in search results. Putting them on different pages without coordinating them is how sites end up with crawlable pages that should be indexed, and disallowed pages that slip into the index anyway.
FAQ
Does robots.txt remove a page from Google’s index?
No. A Disallow prevents crawling, not indexing. If a disallowed URL is reachable from elsewhere on the web, Google can still list it based on those signals. To remove a page from the index entirely, use a noindex meta tag or response header.
Where does the robots.txt file have to live?
It must be served from the root of the host at /robots.txt, with a 200 status and a text/plain content type. RFC 9309 requires each scheme, host, and port combination to have its own file, so a robots.txt on port 8080 governs port 8080 — the port is part of the scope, not restricted to 80 or 443. A file at /subdir/robots.txt is ignored, since only the root path applies.
How long does it take for changes to take effect?
Google caches robots.txt for up to 24 hours, so changes propagate within a day in most cases. Other crawlers vary, and there is no push mechanism, so allow up to a week for a change to be reflected across the major search engines.
Can I use robots.txt to stop scrapers?
Not meaningfully. Polite crawlers, including the major search engines, honor it. Malicious scrapers do not, and they are usually the ones you actually want to stop. Use authentication, rate limiting, or a WAF for anything that genuinely needs protection.
Ship a correct file today
Open your current /robots.txt in a browser tab right now and read it line by line. If you see Disallow: / in a production environment, fix it before you do anything else. If the file lists no sitemap, add one. If it blocks an assets directory, allow it through. Those three checks cover the majority of the broken robots.txt files I have audited. Build a fresh one with the Robots.txt Generator when you are ready to start from a known-good baseline.