robots.txt vs. Meta Robots Tags
Two mechanisms that sound like the same thing and are not. Getting them backwards is the single most common robots.txt misconception.
Crawling and Indexing Are Different Questions
Search engines make two separate decisions about every URL they know about: whether to crawl it — fetch its content by requesting the URL — and whether to index it — store the URL (and whatever it learned about it) so the URL can appear in search results. These sound like the same thing, and in the common case where nothing prevents either, they mostly track together. But each has its own control mechanism, and the two mechanisms don't know about each other:
- robots.txt — a site-wide file that controls crawling only. It cannot remove a URL from the index.
- Meta robots tag — an HTML
<meta>element on a specific page that controls indexing (and other behaviors) for that page. It requires the page to be crawled first. - X-Robots-Tag — the same directives as the meta tag, delivered as an HTTP response header instead. Also requires crawling, and works for non-HTML files where a
<meta>tag isn't possible.
That last constraint — meta robots and X-Robots-Tag both require crawling to take effect — is the source of the most common robots.txt mistake, covered in more depth in our common mistakes guide: if you Disallow a page in robots.txt, Google never fetches it, so it never sees a noindex tag on that page even if you added one. The two mechanisms actively work against each other when combined this way.
Why a Blocked Page Can Still Appear in Search Results
This is the scenario that convinces people robots.txt is broken, when it's working exactly as specified. Suppose /internal-report/ is Disallow'd, but another page on the web — your own site or someone else's — links to it with the anchor text "Q3 Internal Report." Google can see that link during a normal crawl of the linking page. Because robots.txt only prevented Google from fetching /internal-report/ itself, Google may still index the URL based on the link data alone — typically showing a bare result with no snippet, something like "No information is available for this page." The page's content never entered the index; only its existence and the fact that it's linked did.
For genuinely sensitive content, this is a meaningful gap: robots.txt alone does not guarantee a URL stays out of search results.
Which Mechanism Do You Actually Want?
| Goal | Use | Why |
|---|---|---|
| Save crawl budget on low-value URLs (faceted search, internal search results) | robots.txt | You don't care if these are indexed via links — you just don't want crawl effort spent fetching them. |
| Keep a specific page out of search results entirely | Meta robots noindex | Requires the page to stay crawlable so the noindex directive is actually seen. |
| Keep a PDF, image, or API response out of search results | X-Robots-Tag header | There's no HTML to put a <meta> tag in, so the directive travels as a response header instead. |
| Prevent a specific bot (e.g. an AI crawler) from fetching content at all | robots.txt | Advisory only — a compliant bot honors it, but it's the only mechanism that acts before the fetch happens. |
Two mechanisms are occasionally used together correctly: robots.txt to keep crawl budget away from a large low-value section, and meta robots on the small number of pages within it that genuinely must never appear in search results — accepting that those specific pages need to stay crawlable to enforce that.
X-Robots-Tag in Practice
X-Robots-Tag accepts the same directive vocabulary as the meta robots tag — noindex, nofollow, noarchive, none (shorthand for noindex, nofollow), and others — set as an HTTP response header:
X-Robots-Tag: noindex # Target a specific crawler only, same syntax as meta robots X-Robots-Tag: googlebot: noindex
This is set at the web server or application layer (an Nginx add_header directive, an Express middleware, a Next.js API route response header) rather than in markup, which is exactly why it covers non-HTML responses. It's also useful for applying a blanket noindex rule to an entire path prefix — a whole/staging/ environment, for instance — without editing every page's <head> individually. Inspect any URL's response headers (along with DNS, SSL, and other security signals) with our Domain Intelligence Dashboard to confirm an X-Robots-Tag is actually being sent as expected.
The Rule of Thumb
If the question is "should crawlers spend effort fetching this," the answer lives in robots.txt. If the question is "should this specific page be allowed to appear in search results," the answer lives in a meta robots tag or X-Robots-Tag header — and that page must remain crawlable for the answer to take effect at all. Confusing the two, or Disallow-ing a page you actually wanted noindex'd, is the single most common category of robots.txt bug. Build and test your robots.txt rules against real paths with our Robots.txt Generator & Tester — the interactive tester shows exactly which rule matches a path, the same longest-match logic covered in our complete guide, before you rely on it in production.
Further Reading
- Google Search Central — Robots meta tag, data-nosnippet, and X-Robots-Tag
The full directive vocabulary and the authoritative statement that these require the page to be crawlable.
- Google Search Central — Block Search indexing with noindex
Explains precisely why combining Disallow with noindex on the same URL fails, and what to do instead.
- RFC 9309 — Robots Exclusion Protocol
Defines robots.txt as scoped strictly to crawling behavior, not indexing.