robots.txt rules: syntax, matching and common mistakes

robots.txt is a plain-text file at the root of a host that tells crawlers which URL paths they may request. Rules are grouped by user agent, each group lists Allow and Disallow paths, and the most specific matching rule wins. The format is standardized in RFC 9309.

It controls crawling only. It doesn't remove pages from search results, and it isn't access control: compliant crawlers follow it voluntarily, and anyone can still open a blocked URL. The file is public, so listing a secret path in it advertises that path.

Crawlers look for the file only at /robots.txt at the root of a host, such as https://www.example.com/robots.txt. A robots.txt in a subfolder is ignored.

Rules apply to the protocol, host and port the file is served from. The file on example.com doesn't cover www.example.com or shop.example.com, so each subdomain needs its own.

Status codes matter. If the file returns 404 or another 4xx, RFC 9309 allows crawlers to access everything. Under the same RFC, a 5xx or an unreachable file means crawlers must assume the whole site is disallowed.

Crawlers usually cache the file for up to 24 hours, so changes aren't seen at once. Keep it under 500 KiB, the minimum RFC 9309 requires crawlers to read; Google ignores anything past that.

Each line is a field name, a colon and a value, and # starts a comment. A group begins with one or more User-agent lines, followed by the Allow and Disallow rules for those crawlers.

A crawler obeys only one group: the one whose User-agent most specifically matches its name. It falls back to the * group only when no named group matches. User-agent names match without regard to case, but paths are case-sensitive.

An empty Disallow allows everything. Sitemap lines stand outside the groups and take a full URL. In the example below, ExampleBot reads only its own group, so it may crawl /admin/ and /search.

Fields outside the standard, such as Crawl-delay, are ignored by Google, though some other crawlers honor them. Google also ignores noindex lines in robots.txt.

For each URL, the crawler compares its path and query string with every Allow and Disallow rule in its group. The rule with the longest matching path wins. If an Allow and a Disallow rule match with the same length, the Allow rule applies, and a URL that matches no rule is allowed.

Two characters have special meaning. An asterisk (*) matches any sequence of characters, and a dollar sign ($) at the end of a rule anchors it to the end of the URL.

Blocking a URL in robots.txt stops compliant crawlers from fetching it, but the URL can still be indexed if other pages link to it. It may then appear in results with no description, since the crawler never read the page.

A noindex directive, in a robots meta tag or an X-Robots-Tag header, tells search engines to keep a page out of results. They see it only if they can fetch the page. Block a page in robots.txt and add noindex, and the noindex is never read.

To remove a page from search, allow crawling and serve noindex, or return 404 or 410, then wait for the page to be recrawled. For content that must stay private, require a login.

Staging sites often block every crawler with two lines: User-agent: * followed by Disallow: /. When that file reaches production, compliant crawlers stop fetching the live site, new pages aren't discovered and existing results can lose their descriptions.

Check the production robots.txt right after launch and after any deploy that changes configuration. Better still, generate the file per environment and protect staging with a login, which also keeps it out of search results.