Blocks the wp-admin dashboard from crawlers while explicitly allowing admin-ajax.php, which many themes and plugins rely on for front-end AJAX requests.
Keeps /_next/static/ and /_next/image crawlable so Googlebot can fetch the JS/CSS/images it needs to render client-side content — blocking /_next/ entirely is an outdated pattern that breaks rendering-based indexing. Blocks internal API routes, which have no SEO value.
Allows crawling of the entire site. Suitable for a client-rendered SPA where every route should be indexable and there is no admin area or private section to hide.
Blocks the cart, checkout, and account pages, which are user-specific, offer no unique content to index, and can otherwise waste crawl budget or leak into search results.
Allows crawling of all posts and pages while blocking RSS/Atom feed URLs and comment-reply links, which duplicate post content under different URLs and dilute crawl budget.
Blocks every crawler from every path. Use on staging/preview deployments to prevent unfinished or duplicate content from being indexed — remember to remove this before launch.
1User-agent: *2Disallow: /wp-admin/3Allow: /wp-admin/admin-ajax.php4 5Sitemap: https://example.com/sitemap.xml