Robots.txt Generator

A robots.txt file is a plain-text file served at the root of a domain (https://example.com/robots.txt) that follows the Robots Exclusion Protocol originally drafted at robotstxt.org in 1994 and standardised by the IETF as RFC 9309 in 2022. It tells well-behaved crawlers which URL paths they may fetch using two core directives: User-agent (the bot the block applies to) and Disallow (the path prefix to skip).

This generator emits the canonical syntax: each block opens with one or more User-agent lines, followed by Disallow rules and an optional Allow rule that overrides a broader Disallow. Paths are matched as prefixes against the request URL, with two wildcard characters supported by major crawlers — * for any sequence and $ to anchor the end of a URL.

The most important thing to understand is that robots.txt is advisory, not enforced. It is a request, not a firewall. Compliant crawlers (Googlebot, Bingbot, DuckDuckBot, most academic bots) honour it; aggressive scrapers, malware harvesters, and many AI training crawlers either ignore it or read it to find what you are trying to hide. Anything truly sensitive needs server-side authentication, not a Disallow line.

Disallow blocks crawling, but it does not block indexing. If another site links to a Disallow-ed URL, Google can still list it in search results with no snippet — the page is indexed without ever being fetched. To keep a page out of the index entirely, allow crawling and add a meta robots noindex tag or X-Robots-Tag HTTP header instead.

Crawl-delay is the most misunderstood directive in the file. Google has officially ignored it since 2019 — set crawl rate inside Search Console instead. Bing, Yandex, and Seznam still honour Crawl-delay (in seconds between hits), so the directive is only useful if those crawlers matter to you.

The Sitemap directive is independent of any User-agent block and is read by all major engines. Listing your sitemap.xml URL here is the simplest way to register it without using webmaster tools, and you can list multiple sitemaps if you split them by content type or language.

Keep the file under 500 KiB — Google parses only the first 500 KiB and silently drops the rest. Use # for comments, put each directive on its own line, and validate the result in Google Search Console's robots.txt Tester before deploying, since a stray Disallow: / can deindex an entire site within days.