{"canonical_url":"https://engineer.company/glossary/crawler/","content_markdown":"# Crawler\n\n## What it is\n\nThere is no master list of the web's pages, so a search engine has to go and\nfind them, and the program it sends is a crawler. It starts from addresses it\nalready knows, fetches each one over HTTP, reads the links in what comes back\nand adds the new addresses to its list. Google's guide to how Search works gives\nthe other names in use, \"also known as a crawler, robot, bot, or spider\", and\nRFC 9309, the robots exclusion standard, describes crawlers as automated clients\nthat search engines run \"to recursively traverse links for indexing\".\n\nLinks are the main road in. Google says Googlebot finds new addresses \"primarily\nfrom links embedded in previously crawled pages\", and its guide gives the\nfamiliar case of a category page that links to a new post. A sitemap is the\nsecond road, a list the site owner hands over directly. A page with neither\ngives a crawler no way to learn that it exists.\n\nFetching is the first of three stages Google describes. Crawling downloads the\ntext, images and video of a page; indexing analyses what was downloaded and\nstores it; serving answers a search from that index. During the crawl Google\nalso renders the page, running its JavaScript in a recent version of Chrome,\nsince without that step it \"might not see that content\". None of the stages is\npromised: Google \"doesn't guarantee that it will crawl, index, or serve your\npage\", and it \"doesn't accept payment to crawl a site more frequently\".\n\nA crawler is also expected to say who it is. RFC 9309 has each crawler set its\nown name, a product token made only of letters, underscores and hyphens, and\nsays the token should appear in the identification string the crawler sends,\nwhich over HTTP is the User-Agent header, and that the string should describe\nthe crawler's purpose. The RFC's example header carries `ExampleBot/0.1` and a\nlink to a page about the crawler, and the matching robots.txt line reads\n`user-agent: ExampleBot`. The name is a claim, not a credential, since any\nprogram can send it.\n\nNot every automated request from Google is a crawl. Its overview sorts its\nclients into three kinds: common crawlers such as Googlebot, which \"always\nrespect robots.txt rules for automatic crawls\"; special-case crawlers, which\nwork under an agreement with the site and may or may not follow those rules; and\nuser-triggered fetchers, which usually make a single request because a person\nasked, and ignore robots.txt.\n\n## Why it matters\n\nA crawler decides for itself when to come back, and its manners set part of the\ncost of being found. Google says Googlebot \"shouldn't access your site more than\nonce every few seconds on average\" on most sites, and its crawlers back off when\na site struggles; in the Search guide's words, HTTP 500 errors mean \"slow down\".\nGoogle's crawl budget guide describes the whole mechanism, but it is written for\nsites of roughly a million pages changing weekly or ten thousand changing daily,\nand Google says a site whose pages are crawled the day they are published does\nnot need it.\n\nA crawler also reads only so much of each file. Googlebot crawling for Search\ntakes the first 2 MB of any file whose type it supports and the first 64 MB of a\nPDF, measured before compression, and ignores the rest.\n\nThe weak point is the name. Google warns that the User-Agent header Googlebot\nsends \"is often spoofed by other crawlers\", so a log line that says Googlebot\nproves nothing by itself. Google documents two checks. For a single address, run\na reverse DNS lookup, confirm that the host name ends in `googlebot.com`,\n`google.com` or `googleusercontent.com`, then look that name up forwards and\nconfirm it returns the same address. Google's own example:\n\n```text\nhost 66.249.66.1\n1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.\nhost crawl-66-249-66-1.googlebot.com\ncrawl-66-249-66-1.googlebot.com has address 66.249.66.1\n```\n\nFor many addresses at once, match each one against the IP ranges Google\npublishes as JSON files in CIDR notation, one file per kind of client. Both\nchecks rest on the network a request came from, which the sender cannot simply\ntype into a header.\n\n## What it looks like in practice\n\nThis site's server keeps an access log, and the [privacy page](/privacy/) says\nhow crawlers are told apart in it. Each line keeps the user agent, \"which is how\nwe separate people from search-engine crawlers\", and a second step follows: \"A\ncrawler that gives a name is checked against the networks its owner publishes,\nusing the blurred address below.\" That is Google's second method, and it suits\nwhat the log keeps. The last part of every IP address is replaced with zeroes\nbefore the line reaches the disk, so the full address a reverse lookup needs is\nnever stored.\n\nThe summaries treat crawlers as an audience of their own: \"the same for\neverything automated\" beside the thirty days kept for people, a daily tally of\n\"so many pages read, so many of them by crawlers, so many reaching the contact\npage\", and a monthly ranking of \"which crawlers and automated scanners called\".\nA crawler that names itself honestly is counted under its own name, and a famous\nname sent from a network its owner does not publish fails the check.\n\nThe [page on being ready for AI agents](/ai-ready/) starts from the same reader:\n\"An agent starts where a crawler does\", at the robots.txt file, the sitemap and\nthe search indexes. Its answer is to make each of them deliberate, down to an\nIndexNow announcement on every deploy, so the engines that take part \"hear of a\nnew page the day it ships rather than at their next visit\".\n\nDates are one way a site talks to a crawler, and they can end up saying nothing\nuseful. The write-up of [a sitemap that gave 172 of its 176 pages one\ntimestamp](/portfolio/fixed-a-sitemap-with-one-shared-timestamp/) puts the\nstake plainly: \"a modification date is a promise to a crawler that a page's\ncontent changed\". A site that makes the promise for every page at once, it goes\non, \"is either telling the truth about a full rewrite or telling nobody anything\nuseful\".\n\n## Where it goes wrong\n\n**Requests that call themselves Googlebot do not come from Google.** The log\nshows heavy traffic under Googlebot's name, and the owner either lets it through\nas a welcome guest or blocks the name outright. Since other crawlers often copy\nthat name, neither reaction is safe: the first trusts an impostor, and the\nsecond shuts out the real crawler along with it. Check the source network, by\nreverse and forward DNS for one address or against Google's published ranges for\nmany, and decide on that rather than on the name.\n\n**A published page never shows up in a crawl.** Nothing links to it, or it can\nonly be reached after signing in. With no central registry, a crawler learns of\nan address from links and sitemaps, and Google's guide names pages behind a\nlogin among those its crawler cannot reach. Link the page from somewhere already\ncrawled, list it in the sitemap, and keep anything meant for search outside the\nlogin.\n\n**The crawl rate drops after a bad stretch for the server.** Fewer pages are\nfetched each day, and changes take longer to be seen. Slow responses, 5xx errors\nand 429 answers all lower Google's crawl capacity limit, which climbs back only\nas responses become stable and quick, and every Google crawler on the host\nshares what is left. Fix the errors and the latency first; on a site of a few\nthousand pages, Google's crawl budget guide is not the place to start.\n\n**An unlinked page turns up in a crawler's requests.** The address was never\npublished, perhaps a private draft or a test copy, and it still appears in the\naccess log under a crawler's name. Google calls it \"almost impossible to keep a\nsite secret by not publishing links to it\", since a click from that page to\nanother site can leave its address in a referrer log that gets published. Put\nanything private behind password protection, which Google names as the way to\nkeep both crawlers and people out.\n\n## Further reading\n\n- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)\n  — the standard's view of crawlers: product tokens, the User-Agent string and\n  the rules a crawler is asked to honour.\n- [In-depth guide to how Google Search works](https://developers.google.com/search/docs/fundamentals/how-search-works)\n  — discovery, crawling, rendering and indexing as Google describes them.\n- [Overview of Google crawlers and fetchers (user agents)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers)\n  — the three kinds of Google client, the file size limit, protocols and caching.\n- [Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)\n  — the DNS lookups and the published IP ranges, with worked examples.\n- [Optimize your crawl budget](https://developers.google.com/crawling/docs/crawl-budget)\n  — how Google sets its crawl rate, and which sites need to tune it.\n- [Googlebot](https://developers.google.com/search/docs/crawling-indexing/googlebot)\n  — the smartphone and desktop crawlers, how often Googlebot visits, its file\n  size cutoff and why an unlinked page is not secret.\n\n\u003chttps://engineer.company/glossary/crawler/\u003e\n","date_modified":"2026-10-20T21:59:50Z","description":"How a crawler finds pages and paces its visits, how to tell a real one from an impostor in your logs, and why being crawled is not being indexed.","formats":{"gemtext":"https://engineer.company/glossary/crawler/index.gmi","gopher":"https://engineer.company/glossary/crawler/gophermap","html":"https://engineer.company/glossary/crawler/","json":"https://engineer.company/glossary/crawler/index.json","markdown":"https://engineer.company/glossary/crawler/index.md","pdf":"https://engineer.company/glossary/crawler/index.pdf"},"kind":"page","language":"en","title":"What is a crawler?","url":"https://engineer.company/glossary/crawler/","version":1}