# robots.txt ## What it is Every host on the web can publish, at the fixed path `/robots.txt`, a plain text file of requests addressed to the programs that crawl it. Martijn Koster defined the convention in 1994, and in September 2022 the IETF published it as RFC 9309, the Robots Exclusion Protocol, adding precise language for the rules and instructions for errors and caching. The file is a list of groups. A group opens with one or more `user-agent` lines, each naming a crawler by its product token, such as `ExampleBot`, or `*` for every crawler. Below them come `allow` and `disallow` lines, each holding a path. A crawler looks for the group that names its own token, matched without regard to case, and obeys that group; when several groups name it, their rules merge into one. Only a crawler with no group of its own falls back to the `*` group, and one that finds neither has no rules to follow. Within a group, a rule applies when its path matches the start of the address, and the most specific match wins, meaning the one with the most octets. A tie between `allow` and `disallow` goes to `allow`, and an address that no rule matches is allowed. Two characters widen a path: `*` stands for any run of characters and `$` pins the end of the address. A `#` starts a comment, and the file's own path is always allowed. The RFC also fixes how the file is fetched. It sits at that lowercase path at the top of the service, so every host answers for itself and a file on one subdomain says nothing about another. It is UTF-8 plain text. A crawler should follow at least five redirects to reach it, should not use a cached copy for more than 24 hours, and may cap how much it parses, though never below 500 KiB. Lines outside the protocol are tolerated. The RFC calls them other records, gives `Sitemap` as its example, and lets a crawler interpret them or not, on one condition: such a line must never end a group. ## Why it matters The RFC states the limit in its introduction: "These rules are not a form of access authorization." The file is a request. A crawler that honours it stays out, and one that does not reads whatever the server hands it. The security section adds that listing a path in the file makes it public and discoverable, so anything that must stay private needs a real control, such as HTTP authentication. Google's documentation narrows the purpose further. It describes the file as "used mainly to avoid overloading your site with requests" and says "it is not a mechanism for keeping a web page out of Google". A disallowed address can still be indexed when other pages link to it, and then appears in results without a description, because the crawler never read the page. To keep a page out of results, Google points to a `noindex` rule or a password, and a `noindex` rule only works on a page the crawler is allowed to fetch. What the file does well is the job it was written for: steering crawlers away from parts of a site that would cost the server requests and give a search engine little, such as unimportant or near-identical pages. Its reach is also its risk. One line under `*` applies to every address and every crawler that honours the file, and a change can take up to a day to be seen, so `Disallow: /` typed in the wrong group closes a whole site. ## What it looks like in practice This site's file is long, and almost all of it is comments. The part a crawler acts on is four lines: ```text User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Sitemap: https://engineer.company/sitemap.xml ``` One group covers every crawler, and `Allow: /` matches every address. A group with no rules would allow exactly the same, since an address that no rule matches is allowed, but the line is written out because the openness is a choice. The comment above it calls the policy "Decided rather than inherited", and [the page on what AI agents need from a site](/ai-ready/) sets the aim of making the file "state a decision rather than an accident". `Content-Signal` is not a record RFC 9309 defines. It comes from the Content Signals vocabulary, and here it consents to collection for search, for answering a question at inference time and for training. Under the RFC it is one of those other records: a crawler that knows the convention may read it, one that predates it skips the line, as the file's own comment says, and neither may treat it as the end of the group. The `Sitemap:` line is another record of that kind and points at the site's sitemap index. The [credits page](/credits/) sums the file up the same way: it "allows everything, to search crawlers and AI systems alike" and names the sitemap. What the file leaves out was decided too. Among the five recommendations this site declined, the AI readiness page lists rules per agent, keeping one robots rule for every agent "because a rule written for one crawler silently replaces the general one for it." Had the file given one AI crawler a group of its own, that crawler would read only that group, and a `Disallow` added later under `*` would never reach it. The rest is comments, written for whoever opens the file: the reasoning behind the policy, then the addresses of `llms.txt`, `agents.txt`, `humans.txt`, `security.txt` and other machine-readable files, kept as comments "because robots.txt has no field for them". That makes the comments part of what ships. A `#` line in the template is served byte for byte to every crawler, and twice one carried a note meant for a maintainer, pointing at a planning document in a repository that is not public. A check over the built site now fails when a served file carries a path into the repository, and maintainer notes in the template sit in template comments, which the generator removes. ## Where it goes wrong **A rule meant for everyone stops applying to one crawler.** Someone adds a group for a single bot, perhaps to keep it out of one folder, and from then on that bot ignores everything under `*`, including every `Disallow` written there. RFC 9309 sends a crawler to the `*` group only when no group names it, and Google's documentation says specific groups and the `*` group "are not combined". Repeat each general rule inside every named group, or keep a single group, as this site does. **A blocked page still appears in search results.** The address shows up with no description, perhaps with the words other sites used when linking to it. Google never fetched the page, so it knows only that the address exists and what others call it. A `Disallow` controls crawling, not indexing. Remove the rule and serve `noindex` on the page, or put the page behind a password; a `noindex` hidden behind a `Disallow` is never read. **The file advertises what it was meant to hide.** A line such as `Disallow: /admin/` tells every reader, polite crawler or not, exactly where to look, and the file is public by design. The RFC's security section says listing paths makes them discoverable and calls for access control instead. Put private material behind authentication and leave it out of the file entirely. **A server error stops the crawl of a whole site.** The pages load, but the request for `/robots.txt` returns a 5xx status or times out, and a crawler that follows the RFC must treat that as a complete disallow. Google stops crawling for the first 12 hours, then works from its last good copy for up to 30 days. The opposite mistake is quieter: Google treats a 4xx answer other than 429, including 401 and 403, as if no file existed, so every address is open. Serve the file as static text with a 200 status, and monitor `/robots.txt` itself, not only the home page. ## Further reading - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) — the standard itself: groups, matching, error handling, caching and the security note. - [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) — what Google says the file is for, and why it does not hide a page from results. - [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) — status codes, caching, the fields Google ignores and how it picks a group. - [Block Search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing) — the rule that does keep a page out of results, and why it needs the page crawlable.