{"canonical_url":"https://engineer.company/glossary/sitemap/","content_markdown":"# Sitemap\n\n## What it is\n\nA sitemap is a file a website publishes about itself: the addresses its owner\nwants search engines to know, one entry per page, each with a few optional facts\nattached. It exists because a crawler otherwise learns a site the slow way, by\nfollowing links from pages it already knows, and some pages sit at the end of a\nlong chain of links or have hardly any inbound links at all.\n\nThe format the major engines share is the Sitemaps protocol, published at\nsitemaps.org. Its XML version wraps the list in a `\u003curlset\u003e` element with one\n`\u003curl\u003e` per page. Inside it, `\u003cloc\u003e` holds the full address and is the only\nrequired child. `\u003clastmod\u003e` is the date the page last changed, `\u003cchangefreq\u003e` a\nguess at how often it will change, and `\u003cpriority\u003e` a value from 0.0 to 1.0 that\nranks the page against the site's own other pages, never against anyone else's.\nThe protocol also accepts a plain text file with one address per line, or an RSS\nor Atom feed.\n\nThe protocol sets hard edges. One file holds at most 50,000 addresses and 50 MB\nuncompressed, may be gzipped, and must be UTF-8 with its values entity-escaped;\neach address stays under 2,048 characters. A larger site splits its list and\npublishes a sitemap index, a second kind of file that names up to 50,000\nsitemaps on the same site. A sitemap may also list only addresses at or below\nits own location: a file at `/catalog/sitemap.xml` covers the `/catalog/` pages\nand stops at that folder's edge.\n\nA search engine learns where the file lives in one of two ways. A `Sitemap:`\nline in robots.txt gives its full address, independently of any user-agent\ngroup, or the owner submits it through the engine's own tools, such as Google\nSearch Console.\n\nThe same word also names a page of links for human visitors; when the\ndifference matters, people say \"XML sitemap\" for the one written for programs.\n\n## Why it matters\n\nGoogle's documentation is candid about the limits. A site whose pages are\nproperly linked can usually be discovered without a sitemap, and Google draws\nthe line for \"small\" at about 500 pages that should appear in results. It also\nsays a sitemap \"doesn't guarantee that all the items in your sitemap will be\ncrawled and indexed\", and that submitting one \"is merely a hint\". The cases\nwhere it helps most are large sites, new sites with few links from elsewhere,\nand sites heavy with video, images or news.\n\nAlmost every site generator writes one for free, so the real decision is what\nthe file claims, and every entry is a statement an engine can check against the\npage itself.\n\nWhich addresses go in? Google asks for the canonical address of each page. A\nlist padded with redirects, duplicates or pages marked\n`noindex` asks for exactly what the rest of the site says to leave alone.\n\nWhich fields? Google says it ignores `\u003cpriority\u003e` and `\u003cchangefreq\u003e`. It uses\n`\u003clastmod\u003e` only \"if it's consistently and verifiably (for example by comparing\nto the last modification of the page) accurate\", and the protocol says the date\n\"must be set to the date the linked page was last modified, not when the sitemap\nis generated\". Google counts changes to a page's main content, structured data\nor links as significant, and a new copyright year as trivial.\n\nThat is where the cost sits. A date from the file system is free and usually\nwrong, because deploys, exports and fresh checkouts touch files whose words\nstayed the same. A date from version history is honest, but the build then needs\nthat history present and read in the right order. A date typed by hand is\nprecise until someone forgets to change it.\n\n## What it looks like in practice\n\nThis site's robots.txt names one address, `https://engineer.company/sitemap.xml`,\nand that file is a sitemap index naming the sitemaps for the published language\neditions. Each page entry carries the address and a date, plus links to the\nsame page in its other published translations. `\u003cpriority\u003e` and\n`\u003cchangefreq\u003e` are left out, since Google says it ignores both and the protocol\nmakes neither of them required. When this article was written, the entry for\nthe page on AI readiness read like this, its language links set aside:\n\n```xml\n\u003curl\u003e\n  \u003cloc\u003ehttps://engineer.company/ai-ready/\u003c/loc\u003e\n  \u003clastmod\u003e2026-10-15T14:25:16+02:00\u003c/lastmod\u003e\n\u003c/url\u003e\n```\n\nThe list is selective on purpose. The company's focused satellite sites\nrepublish work from here with a canonical pointing back, and the same template\nkeeps those copies out of each satellite's sitemap: an entry asks an engine to\nindex that exact address, and asking for a copy while naming another page as\nthe original would contradict itself. Each host announces only its own pages.\n\nThe dates have a history of their own. At one point\n[the sitemap told search engines that 172 of its 176 pages had changed at the same instant](/portfolio/fixed-a-sitemap-with-one-shared-timestamp/).\nThe cause was ordinary: the date came from each file's modification time, and\nthe content export rewrites every file on every run. The fix moved the source to\nversion history, with a fallback chain that tries an explicit field, then the\ncommit history, then the file. It held one trap worth knowing: the literal field\nname has to appear in that list, or the generator never reads it. Afterwards, a\nre-sync following a month of edits touched eight files of 186 instead of all of\nthem, and the sitemap says something true.\n\nThat date now does work beyond the sitemap. As\n[the page on being found by AI agents](/ai-ready/) describes, every deploy\nannounces what changed through IndexNow, a protocol that Bing and several\nother engines take part in. The deploy script decides what changed by comparing\neach page's `lastmod` with what it last sent, so a deploy that edits one page\nannounces one page. Google is not on IndexNow's own list of participating\nengines, so it learns of the change when it next reads the sitemap. A guard in\nthe build keeps the dates honest: it reads every sitemap date against the\npage's history and refuses to publish when one is wrong. For people rather than\nprograms there is also a [readable index of every page](/sitemap/).\n\n## Where it goes wrong\n\n**Every page claims to have changed today.** The sitemap carries one date, or\none per deploy, across hundreds of pages, and an engine reading it learns\nnothing about which work is new. The cause is a date taken from the file system\nor the build clock. Google uses `\u003clastmod\u003e` only when it is consistently and\nverifiably accurate, which a shared timestamp visibly is not. Take the date from\nthe content's own history, and compare a sample of entries against it after any\nchange to the build.\n\n**The list asks for pages the site wants left out.** Redirects, duplicate\naddresses carrying tracking parameters, pages marked `noindex`, staging hosts and\ncopies whose canonical points elsewhere all appear in sitemaps written by a\ngenerator that lists every file it built. Each one sends a crawler to an address\nthe page itself then disowns. List only the canonical address of each page meant\nfor results, and let the sitemap and the canonical tag answer from one rule, as\nthis site's templates do.\n\n**Entries fall outside what the file may cover.** A sitemap at\n`/blog/sitemap.xml` lists shop pages, or writes relative paths such as\n`/page.html`. The protocol limits a file to addresses at or below its own\nfolder, and Google crawls addresses exactly as written. Publish the sitemap at\nthe root, write every `\u003cloc\u003e` as a complete `https://` address on the same host,\nand switch to a sitemap index before the list nears 50,000 entries or 50 MB.\n\n**Pages listed for months stay out of the index.** The owner reads the sitemap\nas an instruction and waits. Google calls it a hint: a listed address can still\nbe skipped or folded into a duplicate for reasons outside the file. Look at the\npage rather than the list: whether other pages link to it, whether its canonical\nnames itself, and whether robots.txt or a `noindex` rule keeps a crawler away.\n\n## Further reading\n\n- [Sitemaps XML format](https://www.sitemaps.org/protocol.html) — the protocol\n  itself: every tag, the size limits, index files and which addresses a file may\n  list.\n- [Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap)\n  — Google's rules, including the tags it ignores and when it trusts a date.\n- [Learn about sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview)\n  — when Google says a site needs one and when good linking will do.\n- [IndexNow documentation](https://www.indexnow.org/documentation) — the protocol\n  that tells participating engines about a change on the day it ships.\n- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)\n  — the robots.txt standard, which lets a crawler read a `Sitemap:` line beside\n  its rules.\n\n\u003chttps://engineer.company/glossary/sitemap/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"What goes into an XML sitemap, which of its fields Google says it reads, and why each page's date belongs to its words rather than its file.","formats":{"gemtext":"https://engineer.company/glossary/sitemap/index.gmi","gopher":"https://engineer.company/glossary/sitemap/gophermap","html":"https://engineer.company/glossary/sitemap/","json":"https://engineer.company/glossary/sitemap/index.json","markdown":"https://engineer.company/glossary/sitemap/index.md","pdf":"https://engineer.company/glossary/sitemap/index.pdf"},"kind":"page","language":"en","title":"What is a sitemap?","url":"https://engineer.company/glossary/sitemap/","version":1}