Robots.txt Generator
Create crawler access rules and add your XML sitemap location with a straightforward robots.txt builder.
Generate a search-engine-friendly robots.txt file
A robots.txt file gives search crawlers instructions about areas they may access. It can also point crawlers to your XML sitemap.
Crawler directives are not access controls. Protect private pages with proper authentication rather than relying on robots.txt.