Robots.txt Generator
Generate a valid robots.txt in seconds. Choose which crawlers to allow or block, set disallowed paths, add a crawl delay, and point bots to your sitemap — then copy the result.
User-agent: * Disallow: /admin Disallow: /cart Disallow: /checkout Disallow: /account Sitemap: https://example.com/sitemap.xml
What robots.txt controls
robots.txt is the first file most crawlers request. It tells well-behaved bots which paths they may crawl, which to skip, and where your sitemap lives. Used correctly it keeps crawl budget on your important pages and out of admin, cart, and search-results URLs.
Built for modern crawlers
Beyond Googlebot and Bingbot, this generator lets you write rules for AI crawlers like GPTBot, ClaudeBot, and PerplexityBot — useful if you want to allow or restrict how your content is used by answer engines.
Handle with care
robots.txt is a crawling directive, not a security or privacy control — disallowed URLs can still be indexed if linked elsewhere, and the file is publicly visible. To keep a page out of search results entirely, use a noindex meta tag or proper authentication instead.
Frequently asked questions
robots.txt belongs at the root of your domain, exactly at https://yourdomain.com/robots.txt. Crawlers only check that location.
No, Disallow does not remove a page from Google. It stops crawling, but a blocked URL can still appear in results when other sites link to it. To drop a page from the index, allow crawling and add a noindex tag, or require authentication.
Yes, you can block AI crawlers like GPTBot. Add a rule targeting the crawler's user-agent, such as GPTBot, ClaudeBot, or PerplexityBot, with the paths you want to disallow. Compliance depends on the crawler honoring robots.txt.
Yes, include your sitemap. A Sitemap directive helps search engines discover all your URLs, and you can list more than one.
Two lines block every compliant crawler: "User-agent: *" followed by "Disallow: /". Use it for staging and preview environments, never on a production site you want indexed, and remember pages already in the index need a noindex tag to drop out.
A typical WordPress robots.txt disallows /wp-admin/ while allowing /wp-admin/admin-ajax.php, which front-end plugins call, and lists your sitemap URL. Blocking /wp-content/ or /wp-includes/ is outdated advice: it stops Google from loading your CSS and JavaScript, which hurts rendering.
Stop doing SEO by hand
These tools handle one step. Essel runs the whole pipeline — it researches, writes, optimizes, and publishes SEO content autonomously, so you don't have to.
Try Essel free