The Complete Guide to robots.txt

What robots.txt Actually Is

robots.txt is a plain-text file served from the root of a site — https://example.com/robots.txt, nowhere else — that tells crawlers which parts of the site they may fetch. It was an informal convention for nearly 30 years before RFC 9309 standardized it in 2022, codifying what Google, Bing, and other major crawlers had already converged on. The core fact that trips people up most: robots.txt is advisory. It asks well-behaved crawlers not to fetch certain paths; it does not — cannot — stop a browser, a scraper, or a malicious bot from requesting them anyway. It is not a security control and it is not the same thing as removing a page from search results (more on that distinction below, and in full in our robots.txt vs. meta robots article).

Every Directive, Explained

User-agent

Starts a group. The value is a crawler's product token — a short name like Googlebot, Bingbot, or GPTBot — not the full HTTP User-Agent header string. The wildcard * matches any crawler that doesn't have its own explicit group. One or more consecutive User-agent lines followed by rules form a single group; a comment or blank line does not end a group, but a new User-agent line appearing after rules have already started does.

Allow / Disallow

Each takes a path pattern. Disallow: /admin/ blocks that path and everything under it; Disallow: with an empty value is a documented no-op (allows everything) — a common source of confusion when someone means to block the whole site and forgets the slash. An empty Disallow: line does nothing; you need Disallow: / to block everything. Two wildcards are supported in path values: * matches any sequence of characters (including none), and a trailing $ anchors the match to the end of the path. /*.pdf$ matches any path ending in .pdf; without the $ it would also match /report.pdf.backup.

Sitemap

An absolute URL to an XML sitemap. Unlike Allow/Disallow, a Sitemap directive is not scoped to any group — it applies site-wide regardless of where in the file it appears, and a file can list multiple. Most sites put it at the very end, but that's convention, not requirement.

Crawl-delay

A non-standard but widely supported directive requesting a minimum number of seconds between requests. Bing, Yandex, and most other crawlers honor it; Googlebot ignores it entirely — Google manages crawl rate through Search Console settings instead. If you're trying to slow down Googlebot specifically, Crawl-delay is the wrong tool.

Host (non-standard)

Originally a Yandex-specific extension for declaring a preferred mirror domain. It is not part of RFC 9309 and is ignored by Googlebot and most other crawlers; including it is harmless but rarely does anything useful today.

How Group Matching and Precedence Work

When a crawler requests a path, it first picks which group applies: if any group names its exact product token, that group (or groups, if the token appears more than once) applies; otherwise, groups naming * apply. A crawler with no matching group at all — no exact match and no wildcard group — is unrestricted for that file.

Within the applicable group(s), every Allow and Disallow rule whose pattern matches the path is a candidate. RFC 9309's tie-breaker is longest match wins — the rule with the more specific (longer) path pattern takes precedence, regardless of whether it's Allow or Disallow, or which order the rules appear in the file. If two matching rules are exactly the same length, Allow wins over Disallow. This is precisely how the classic WordPress pattern works:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

A request for /wp-admin/admin-ajax.php matches both rules, but /wp-admin/admin-ajax.php (28 characters) is longer than /wp-admin/ (10 characters), so Allow wins even though Disallow is listed first and would "normally" read as taking priority. Rule order in the file is irrelevant to the outcome — only pattern length (and the allow-wins tie-break) decides.

Crawl Budget: Why This Matters Beyond Privacy

Search engines allocate a finite amount of crawling effort — crawl budget — to each site, roughly proportional to the site's size, authority, and how fast it responds. Every URL a crawler fetches, including ones with no SEO value, spends part of that budget. Faceted navigation (?sort=, ?color=), internal search result pages, session-ID URLs, and infinite calendar pages are classic budget sinks: they generate enormous numbers of low-value URLs that crowd out crawling of pages you actually want indexed. Blocking these paths in robots.txt is one of the few crawl-budget levers you have direct, immediate control over — it doesn't affect a small site much, but on a site with millions of URLs it can meaningfully redirect crawl effort toward pages that matter.

The opposite mistake — over-blocking — has a cost too. Blocking /_next/static/, a CSS directory, or any path a renderer needs breaks Google's ability to render the page for indexing purposes, which can hurt rankings even though the page itself was never "hidden." See our common mistakes guide for the full list of what not to block.

Test It Yourself

Paste any path below and pick a crawler to see exactly which rule decides the outcome, against a realistic example file (WordPress admin paths blocked with an admin-ajax exception, a faceted-search block, and a full GPTBot block — a common pattern for sites that don't want their content used for AI training). The reasoning shown is the same longest-match logic described above.

For your own site, build and test the full file — including sitemap entries and per-crawler groups — with our Robots.txt Generator & Tester. If you need to inspect the path segments of URLs you're writing patterns for (query strings, encoded characters), the URL Encoder/Decoder breaks a URL down component by component.

A Minimal Checklist

• File is served at exactly /robots.txt, plain text, UTF-8.
• Every group starts with one or more User-agent lines before any rule.
• At least one Sitemap directive, listing your full sitemap URL(s).
• No accidental Disallow: / under the wildcard group in production.
• Rendering-critical assets (JS, CSS, fonts) are not blocked.
• Tested against the paths that matter most, not just skimmed by eye.