{"canonical_url":"https://engineer.company/glossary/robots-txt/","content_markdown":"# robots.txt\n\n## What it is\n\nEvery host on the web can publish, at the fixed path `/robots.txt`, a plain text\nfile of requests addressed to the programs that crawl it. Martijn Koster defined\nthe convention in 1994, and in September 2022 the IETF published it as RFC 9309,\nthe Robots Exclusion Protocol, adding precise language for the rules and\ninstructions for errors and caching.\n\nThe file is a list of groups. A group opens with one or more `user-agent` lines,\neach naming a crawler by its product token, such as `ExampleBot`, or `*` for\nevery crawler. Below them come `allow` and `disallow` lines, each holding a\npath. A crawler looks for the group that names its own token, matched without\nregard to case, and obeys that group; when several groups name it, their rules\nmerge into one. Only a crawler with no group of its own falls back to the `*`\ngroup, and one that finds neither has no rules to follow.\n\nWithin a group, a rule applies when its path matches the start of the address,\nand the most specific match wins, meaning the one with the most octets. A tie\nbetween `allow` and `disallow` goes to `allow`, and an address that no rule\nmatches is allowed. Two characters widen a path: `*` stands for any run of\ncharacters and `$` pins the end of the address. A `#` starts a comment, and the\nfile's own path is always allowed.\n\nThe RFC also fixes how the file is fetched. It sits at that lowercase path at\nthe top of the service, so every host answers for itself and a file on one\nsubdomain says nothing about another. It is UTF-8 plain text. A crawler should\nfollow at least five redirects to reach it, should not use a cached copy for\nmore than 24 hours, and may cap how much it parses, though never below 500 KiB.\n\nLines outside the protocol are tolerated. The RFC calls them other records,\ngives `Sitemap` as its example, and lets a crawler interpret them or not, on one\ncondition: such a line must never end a group.\n\n## Why it matters\n\nThe RFC states the limit in its introduction: \"These rules are not a form of\naccess authorization.\" The file is a request. A crawler that honours it stays\nout, and one that does not reads whatever the server hands it. The security\nsection adds that listing a path in the file makes it public and discoverable,\nso anything that must stay private needs a real control, such as HTTP\nauthentication.\n\nGoogle's documentation narrows the purpose further. It describes the file as\n\"used mainly to avoid overloading your site with requests\" and says \"it is not a\nmechanism for keeping a web page out of Google\". A disallowed address can still\nbe indexed when other pages link to it, and then appears in results without a\ndescription, because the crawler never read the page. To keep a page out of\nresults, Google points to a `noindex` rule or a password, and a `noindex` rule\nonly works on a page the crawler is allowed to fetch.\n\nWhat the file does well is the job it was written for: steering crawlers away\nfrom parts of a site that would cost the server requests and give a search\nengine little, such as unimportant or near-identical pages. Its reach is also\nits risk. One line under `*` applies to every address and every crawler that\nhonours the file, and a change can take up to a day to be seen, so `Disallow: /`\ntyped in the wrong group closes a whole site.\n\n## What it looks like in practice\n\nThis site's file is long, and almost all of it is comments. The part a crawler\nacts on is four lines:\n\n```text\nUser-agent: *\nContent-Signal: search=yes, ai-input=yes, ai-train=yes\nAllow: /\n\nSitemap: https://engineer.company/sitemap.xml\n```\n\nOne group covers every crawler, and `Allow: /` matches every address. A group\nwith no rules would allow exactly the same, since an address that no rule\nmatches is allowed, but the line is written out because the openness is a\nchoice. The comment above it calls the policy \"Decided rather than inherited\",\nand [the page on what AI agents need from a site](/ai-ready/) describes\ncrawl policy as one of the site’s deliberate decisions.\n\n`Content-Signal` is not a record RFC 9309 defines. It comes from the Content\nSignals vocabulary, and here it consents to collection for search, for answering\na question at inference time and for training. Under the RFC it is one of those\nother records: a crawler that knows the convention may read it, one that\npredates it skips the line, as the file's own comment says, and neither may\ntreat it as the end of the group. The `Sitemap:` line is another record of that\nkind and points at the site's sitemap index. The [credits page](/credits/) sums\nthe file up the same way: it \"allows everything, to search crawlers and AI\nsystems alike\" and names the sitemap.\n\nWhat the file leaves out was decided too. Among the five recommendations this\nsite declined, the AI readiness page lists rules per agent, keeping one robots\nrule for every agent \"because a rule written for one crawler silently replaces\nthe general one for it.\" Had the file given one AI crawler a group of its own,\nthat crawler would read only that group, and a `Disallow` added later under `*`\nwould never reach it.\n\nThe rest is comments, written for whoever opens the file: the reasoning behind\nthe policy, then the addresses of `llms.txt`, `agents.txt`, `humans.txt`,\n`security.txt` and other machine-readable files, kept as comments \"because\nrobots.txt has no field for them\". That makes the comments part of what ships. A\n`#` line in the template is served byte for byte to every crawler, and twice one\ncarried a note meant for a maintainer, pointing at a planning document in a\nrepository that is not public. A check over the built site now fails when a\nserved file carries a path into the repository, and maintainer notes in the\ntemplate sit in template comments, which the generator removes.\n\n## Where it goes wrong\n\n**A rule meant for everyone stops applying to one crawler.** Someone adds a\ngroup for a single bot, perhaps to keep it out of one folder, and from then on\nthat bot ignores everything under `*`, including every `Disallow` written there.\nRFC 9309 sends a crawler to the `*` group only when no group names it, and\nGoogle's documentation says specific groups and the `*` group \"are not\ncombined\". Repeat each general rule inside every named group, or keep a single\ngroup, as this site does.\n\n**A blocked page still appears in search results.** The address shows up with no\ndescription, perhaps with the words other sites used when linking to it. Google\nnever fetched the page, so it knows only that the address exists and what others\ncall it. A `Disallow` controls crawling, not indexing. Remove the rule and serve\n`noindex` on the page, or put the page behind a password; a `noindex` hidden\nbehind a `Disallow` is never read.\n\n**The file advertises what it was meant to hide.** A line such as `Disallow:\n/admin/` tells every reader, polite crawler or not, exactly where to look, and\nthe file is public by design. The RFC's security section says listing paths\nmakes them discoverable and calls for access control instead. Put private\nmaterial behind authentication and leave it out of the file entirely.\n\n**A server error stops the crawl of a whole site.** The pages load, but the\nrequest for `/robots.txt` returns a 5xx status or times out, and a crawler that\nfollows the RFC must treat that as a complete disallow. Google stops crawling\nfor the first 12 hours, then works from its last good copy for up to 30 days.\nThe opposite mistake is quieter: Google treats a 4xx answer other than 429,\nincluding 401 and 403, as if no file existed, so every address is open. Serve\nthe file as static text with a 200 status, and monitor `/robots.txt` itself, not\nonly the home page.\n\n## Further reading\n\n- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)\n  — the standard itself: groups, matching, error handling, caching and the\n  security note.\n- [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro)\n  — what Google says the file is for, and why it does not hide a page from\n  results.\n- [How Google interprets the robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)\n  — status codes, caching, the fields Google ignores and how it picks a group.\n- [Block Search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing)\n  — the rule that does keep a page out of results, and why it needs the page\n  crawlable.\n\n\u003chttps://engineer.company/glossary/robots-txt/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"How crawlers read a robots.txt file rule by rule, what it cannot keep out of search results, and why its status code can outweigh its rules.","formats":{"gemtext":"https://engineer.company/glossary/robots-txt/index.gmi","gopher":"https://engineer.company/glossary/robots-txt/gophermap","html":"https://engineer.company/glossary/robots-txt/","json":"https://engineer.company/glossary/robots-txt/index.json","markdown":"https://engineer.company/glossary/robots-txt/index.md","pdf":"https://engineer.company/glossary/robots-txt/index.pdf"},"kind":"page","language":"en","title":"What is robots.txt?","url":"https://engineer.company/glossary/robots-txt/","version":1}