Robots.txt Generator

What is a robots.txt File?

The robots.txt file is a plain-text file placed at the root of your website that gives instructions to search engine crawlers about which pages and directories they are allowed to access. It is the first file most crawlers request when they visit your site, and it plays an important role in controlling how your site is crawled and indexed.

Our Robots.txt Generator builds a correctly formatted file for you. Specify the user-agent, add Allow and Disallow rules for specific paths, optionally set a crawl-delay, and include the URL of your XML sitemap so crawlers can find all your pages.

How to Generate a robots.txt File

• Use a preset to allow or block all crawlers as a starting point
• Set the User-agent (* targets all crawlers)
• Add Allow and Disallow rules for the paths you want to control
• Optionally set a crawl-delay and add your sitemap URL
• Copy the file and upload it to the root of your domain as robots.txt

Understanding the Directives

User-agent specifies which crawler the following rules apply to; an asterisk (*) targets all of them. Disallow blocks a path from being crawled, while Allow explicitly permits one (useful for carving out exceptions inside a disallowed directory). Crawl-delay asks crawlers to wait a set number of seconds between requests, which can ease server load.

The Sitemap directive points crawlers to your XML sitemap, helping them discover and index all of your important pages efficiently. It is good practice to always include it.

Important Cautions

robots.txt controls crawling, not indexing or security. A page blocked in robots.txt can still appear in search results if other sites link to it, and the file is publicly visible to anyone — so never use it to hide sensitive URLs. For true privacy use authentication or the noindex meta tag. Also be careful: a single misplaced Disallow: / can accidentally block your entire site from search engines.