Free technical SEO utility
Build the file before you upload it.
Turn a few intentional crawl rules into a copyable robots.txt draft. Your paths stay in this browser.
1 · Set the rules
Keep it small and specific.
Before you publish
Crawl control is not access control.
A robots.txt file tells participating crawlers which paths they may request. It is useful for managing crawling, not for hiding sensitive pages or guaranteeing that a URL disappears from search results.
- Keep pages and assets that should appear in search accessible to crawlers.
- Use a page-level
noindex, login, or another appropriate control when the aim is to keep content out of search. - Put the final UTF-8 text file at your site’s root:
https://your-domain.example/robots.txt.
Boundary
A draft, not a crawler test.
This generator creates a simple general-crawler group using Allow, Disallow, and an optional sitemap reference. It does not fetch your site, validate a CMS setting, decide whether a rule is safe, or test how a particular crawler will interpret the final file.
Robots exclusion rules are not a security control: paths listed in the file are public, and crawler support or interpretation can vary. Review your deployed file and test it with the tools relevant to your stack.