🤖
Robots.txt Generator
GeneratorGenerate a robots.txt file with custom crawl rules for search engine bots.
About This Tool
robots.txt is the first file search engine crawlers fetch. It tells them which pages not to crawl. Note: robots.txt is publicly visible and tells potential attackers which paths exist. For truly private pages, use authentication — not just robots.txt.