Common robots.txt Mistakes

Check Your File First

Paste your robots.txt below to check it against the same rules this article walks through: malformed directives, contradictory rules, invalid sitemap URLs, and the questionable patterns covered below. The sample loaded by default demonstrates the single most damaging mistake on this list.

1. Blocking CSS, JS, or Other Rendering-Critical Assets

This is the mistake that does the most damage for the least obvious reason. Modern Google indexing renders pages — it runs your JavaScript and applies your CSS to see what a user would actually see, then indexes that rendered content. A Disallow: /*.css or Disallow: /*.js line, or a blanket Disallow: /wp-content/themes/ that happens to catch a site's stylesheets, doesn't just hide those files — it prevents Googlebot from rendering the page correctly at all. The practical effect ranges from missing content in the index to pages being judged low-quality because they render as broken or blank. This is also the reason blanket Disallow: /_next/ patterns are outdated for Next.js sites: /_next/static/ and /_next/image need to stay crawlable, even though blocking /_next/ looks harmless at a glance.

Fix: never block a file extension pattern (*.css, *.js) wholesale. If you must block a build directory, explicitly allow the asset subpaths inside it first — Allow rules can carve out exceptions from a broader Disallow, and per the longest-match rule described in our complete guide, a more specific Allow beats a shorter Disallow regardless of order.

2. Wildcard Misuse: Too Broad or Silently Wrong

The two robots.txt wildcards — * (any sequence, including none) and a trailing $ (end-of-path anchor) — are easy to reach for and easy to get subtly wrong. Two recurring patterns:

• Forgetting the anchor. Disallow: /*.pdf (no $) matches /report.pdf, but it also matches /report.pdf.html or /pdf-viewer/index — anything containing .pdf anywhere in the path, not just files ending in it. Add $ when you mean "ends with."
• Over-broad prefixes. Disallow: /product blocks /product, /products, /product-guide, and anything else starting with those eight characters, because Disallow patterns are prefix matches by default — there's no implicit word boundary. If you mean only the /product/ directory, include the trailing slash: Disallow: /product/.

Both mistakes are invisible by inspection — the file looks reasonable — and only show up when you actually test a representative set of paths, which is why the URL Tester exists as a first-class tab in our tool rather than an afterthought.

3. Treating Disallow as noindex

This is the most common conceptual error, and it's the reason a "blocked" page can still show up in Google search results. robots.txt controls crawling — whether Googlebot fetches a URL's content. It has no effect on indexing — whether that URL's existence and any information about it (its URL, anchor text from links pointing to it, sometimes a description) can appear in search results. If other pages link to a Disallow'd URL, Google can still index the URL itself, typically showing it with a snippet like "No information is available for this page" because it was never allowed to fetch the content to generate a real description.

Sites trying to keep pages out of search results entirely need a meta name="robots" content="noindex" tag (or an X-Robots-Tag HTTP header for non-HTML files) on the page itself — which requires the page to be crawlable, the opposite of what Disallow does. The full mechanics of this interaction, including why you sometimes need both directives and sometimes need to avoid combining them, are in robots.txt vs. meta robots.

4. Forgetting the Sitemap Directive (or Getting the URL Wrong)

A missing Sitemap: line isn't a functional bug — crawlers will still discover pages through links — but it's a free, zero-risk signal that costs nothing to include and speeds up discovery of new and updated URLs. The two failure modes here are simple: forgetting the directive entirely, and pointing it at a relative path (Sitemap: /sitemap.xml) instead of a full absolute URL, which the RFC requires and which our validator flags as a warning. Unlike Allow/Disallow, Sitemap directives apply regardless of which group they're listed under (or whether they're under any group at all), so there's no reason not to list every sitemap a site has, including sitemap index files.

Catching These Before They Ship

All four mistakes share a trait: they're syntactically valid robots.txt, so nothing breaks loudly. The file loads, crawlers parse it without error, and the damage — un-rendered pages, unintentionally blocked sections, a page indexed with no description, slower sitemap discovery — shows up downstream, sometimes weeks later in Search Console. Two habits catch them early: run new or changed robots.txt files through a validator (like the one above) as part of a deploy checklist, and test the specific paths you care about — homepage, key landing pages, checkout, admin — rather than eyeballing the rules and assuming they do what they look like they do.