Robots.txt Generator

What the Robots.txt Generator does

Robots.txt Generator: generates robots.txt from Preset, Sitemap URL and Extra paths to disallow (one per line). Example: preset Allow everything, sitemap URL https://example.com/sitemap.xml and crawl delay None (recommended) gives robots.txt User-agent: * Allow: / Disallow: /admin/ Disallow: /tmp/ Sitemap: https://example.com/sitemap.xml.

The inputs are Preset, Sitemap URL, Extra paths to disallow (one per line) and Crawl delay; the output is robots.txt. Values update as you type, so comparing options is a matter of editing one field.

Typical users are anyone tuning a snippet. What Google displays is rarely exactly what you wrote, and the difference costs clicks.

Analysis is local, so no copy of your page content is retained anywhere.

The field reference, the method and a worked example using the default inputs follow the tool itself.

What do the Robots.txt Generator fields mean?

The Robots.txt Generator uses 4 inputs. Sensible defaults are loaded up front, which means you can change one number at a time instead of filling the whole form first.

How does the Robots.txt Generator work?

Each output is derived from the inputs above in a single pass; there is no hidden state carried between runs, so the same inputs always give the same generated output.

If something is off you are told which field, not given a generic failure.

Worked example

These are the values the Robots.txt Generator loads by default, and the result it produces from them. These are the exact values loaded when the page opens, and the answer they produce.

Robots.txt controls crawling. It does not control indexing.

The single most common robots.txt mistake is treating Disallow as a way to keep a URL out of Google’s index. It is not, and the two ideas get confused constantly because the visible outcomes often look similar.

Worth knowing

• A URL disallowed in robots.txt can still appear in Google’s results, typically as a bare link with "No information is available for this page" instead of a snippet, if other sites link to it, because Google respects the crawl block without ever fetching the page to see any noindex tag it might carry.
• To reliably remove a page from the index: allow crawling and add a noindex meta tag or header, or remove the page and let it 404/410. Robots.txt alone accomplishes neither goal by itself.
• Accidentally disallowing a CSS or JavaScript directory is a specifically well-documented mistake. Google renders pages to evaluate them, and blocking the assets needed to render can make a page look broken or incomplete during that rendering pass.
• Googlebot ignores the non-standard Crawl-delay directive entirely; Bing and Yandex do honour it. Control Google’s crawl rate from Search Console instead.

How to use it

1. Pick a value for Preset: the options are Allow everything, WordPress, E-commerce (block cart, checkout, search) or Block everything (staging).
2. Fill in Sitemap URL.
3. Paste your content into Extra paths to disallow (one per line).
4. Pick a value for Crawl delay: the options are None (recommended), 1 second, 5 seconds or 10 seconds.
5. There is nothing to click. The answer keeps pace with your edits.
6. Read robots.txt in the results panel.
7. Print gives a tidy version without the navigation, which is handy for attaching to a file.

What this tool does not do

• This inspects the markup you supply; it cannot see server headers or your live rankings.
• Search engines weigh hundreds of signals. No single on-page check predicts position.
• Comparisons across tools can differ where each uses a slightly different definition.

Frequently asked questions

Related tools