Robots.txt Generator
Create a correct robots.txt file: allow or block paths, add your sitemap and optionally block AI crawlers like GPTBot. Free generator.
Google ignores Crawl-delay; Bing and Yandex respect it.
robots.txt
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search
Upload it to the root of your site so it's at https://yoursite.com/robots.txt. robots.txt asks bots politely — it doesn’t hide pages from people.
Ad space · responsive
How to use Robots.txt Generator
- Choose what to allow and list paths bots shouldn't crawl.
- Add your sitemap URL, and optionally block AI training crawlers.
- Download robots.txt and upload it to your site's root folder.
Good to know
- Runs in your browser; nothing is uploaded.
Privacy
Runs entirely in your browser. Nothing you enter or open here is uploaded.
Questions
Does robots.txt hide a page from Google?
Not reliably — a blocked page can still be indexed if other sites link to it. To keep a page out of results, use a noindex meta tag instead (and don't block it in robots.txt, or Google can't see the tag).
Does blocking AI crawlers work?
The big AI companies say their crawlers obey robots.txt, but it's a request, not a lock.