Robots.txt Generator
Robots.txt Generator helps you build a correct robots.txt file with allow/disallow rules and a sitemap reference.
How to use
- Choose default crawl behaviour.
- Add disallowed paths and your sitemap.
- Copy the generated robots.txt.
Features
- Allow / disallow rules
- Crawl-delay option
- Sitemap line
- Copy-ready output
Frequently asked questions
Where does robots.txt go?
It must be placed at the root of your domain, e.g. https://example.com/robots.txt.