Every public website should have a robots.txt at the domain root. This single text file tells search-engine crawlers which paths they may visit and which they should skip, and provides links to your sitemap. Getting it right matters because mistakes can either expose pages you intended to hide (those pages still get crawled even if you intended to disallow them, since the disallow line itself is publicly readable) or accidentally block your entire site from being indexed (Disallow: / on User-agent: * is the classic foot-gun). This builder lets you assemble a robots.txt from a structured form: choose user agents (asterisk for everyone, or specific bots like Googlebot, Bingbot, GPTBot, ClaudeBot), add allow and disallow rules per agent, attach one or more sitemap references, and the tool generates a clean file you can copy or download. The output follows the conventions that all major crawlers respect.
Frequently asked questions
All crawlers will assume they can crawl everything that is publicly linked. For most sites this is the right default. You only need a robots.txt if you want to block specific paths from crawling, request crawl-delay throttling, or point crawlers at your sitemap.
Related tools
- Keyword Density CheckerPaste any text to see word counts, the top single words, and the top two-word and three-word phrases.
- Google SERP PreviewVisualize how your page title and meta description will appear in a Google search result.
- OpenGraph PreviewPreview how your page will appear when shared on Facebook, LinkedIn, Slack, and other OG-aware platforms.