Robots.txt Generator

What is a robots.txt File?

The robots.txt file is a plain-text file placed at the root of your website that gives instructions to search engine crawlers about which pages and directories they are allowed to access. It is the first file most crawlers request when they visit your site, and it plays an important role in controlling how your site is crawled and indexed.

Our Robots.txt Generator builds a correctly formatted file for you. Specify the user-agent, add Allow and Disallow rules for specific paths, optionally set a crawl-delay, and include the URL of your XML sitemap so crawlers can find all your pages.

How to Generate a robots.txt File

• Use a preset to allow or block all crawlers as a starting point
• Set the User-agent (* targets all crawlers)
• Add Allow and Disallow rules for the paths you want to control
• Optionally set a crawl-delay and add your sitemap URL
• Copy the file and upload it to the root of your domain as robots.txt

Understanding the Directives

User-agent specifies which crawler the following rules apply to; an asterisk (*) targets all of them. Disallow blocks a path from being crawled, while Allow explicitly permits one (useful for carving out exceptions inside a disallowed directory). Crawl-delay asks crawlers to wait a set number of seconds between requests, which can ease server load.

The Sitemap directive points crawlers to your XML sitemap, helping them discover and index all of your important pages efficiently. It is good practice to always include it.

Important Cautions

robots.txt controls crawling, not indexing or security. A page blocked in robots.txt can still appear in search results if other sites link to it, and the file is publicly visible to anyone — so never use it to hide sensitive URLs. For true privacy use authentication or the noindex meta tag. Also be careful: a single misplaced Disallow: / can accidentally block your entire site from search engines.

Crawl Budget and Why robots.txt Matters

Search engines allocate a limited amount of crawling effort to each site, often called the crawl budget. On large sites with thousands of pages, you do not want crawlers wasting that budget on low-value URLs — internal search results, faceted filter combinations, admin areas, or duplicate parameter variations — while important content waits to be discovered. A well-crafted robots.txt steers crawlers away from the chaff so they spend their time on the pages that actually matter for your rankings.

It is also the natural home for your sitemap reference. Listing your XML sitemap in robots.txt is one of the most reliable ways for crawlers to discover the full structure of your site, especially for new or deep pages that have few internal links pointing to them. Used together — guiding crawlers away from noise and toward your sitemap — robots.txt becomes a quiet but meaningful part of technical SEO rather than just a blocklist.

Why Choose Our Robots.txt Generator?

This generator is completely free, needs no signup, and runs entirely in your browser. It builds a correctly formatted robots.txt from simple inputs and presets, which matters because the syntax is unforgiving — a single stray slash in a Disallow rule can hide your entire site from search engines. Starting from a valid template and adding rules deliberately removes that risk.

The Robots.txt Generator is one of more than 200 free tools on ToopTools. If you manage websites and their SEO, you can pin it to your personalized My Workspace workspace so it sits beside your other SEO and web utilities, always one click away. It works on any modern browser, on desktop and mobile.

Is the robots.txt generator free?

Yes. It is completely free with no limits and no signup. You can build as many robots.txt files as you like, as often as you like, without ever creating an account.

Where do I put the robots.txt file?

It must live at the root of your domain, reachable at yourdomain.com/robots.txt. Crawlers only look for it in that exact location, so a robots.txt placed in a subdirectory will be ignored. Upload the generated file to your site's root directory with the filename robots.txt in lowercase.

Does robots.txt keep a page out of Google?

Not reliably. robots.txt prevents crawling, but a disallowed page can still appear in results if other sites link to it, since Google can index the URL without reading the page. To truly keep a page out of search, allow it to be crawled and add a noindex meta tag, or protect it behind authentication.

What is the difference between Allow and Disallow?

Disallow blocks a path from being crawled, while Allow explicitly permits one. Allow is most useful for carving out an exception inside a blocked directory — for example, disallowing an entire folder but allowing one specific file within it. Together they give you fine-grained control over exactly which paths crawlers may access.

Should I include my sitemap?

Yes, it is good practice. Adding a Sitemap directive that points to your XML sitemap helps crawlers discover all of your important pages efficiently, including deep pages with few internal links. It is one line, costs nothing, and improves how thoroughly and quickly your site is indexed.

Tips for a Safe robots.txt

• Double-check for an accidental Disallow: / that would block your whole site
• Never rely on robots.txt to hide sensitive URLs — the file is public
• Use noindex or authentication when you truly need a page kept private
• Always include a Sitemap directive pointing to your XML sitemap
• Test your file with a robots.txt tester before relying on it

Key Features

• Presets to allow or block all crawlers as a starting point
• Custom Allow and Disallow rules per user-agent
• Optional crawl-delay setting
• Sitemap directive support
• Correctly formatted, ready-to-upload output
• 100% free and browser-based — no signup required

How do I block a page or folder from search engines?

Add a Disallow rule for the path you want to block — for example, Disallow: /admin/ — and the tool builds the robots.txt directive instantly. Place the file at your domain's root so crawlers read it. Note that this asks crawlers not to crawl, rather than hiding a page.

Does the robots.txt generator work offline?

Yes. The file is generated entirely in your browser, so once the page has loaded the tool works without an internet connection and nothing you enter is uploaded. You can build your robots.txt rules privately on your own device.