Sitemap

a list of a site's pages, written for machines

What it is

A sitemap is a file a website publishes about itself: the addresses its owner wants search engines to know, one entry per page, each with a few optional facts attached. It exists because a crawler otherwise learns a site the slow way, by following links from pages it already knows, and some pages sit at the end of a long chain of links or have hardly any inbound links at all.

The format the major engines share is the Sitemaps protocol, published at sitemaps.org. Its XML version wraps the list in a <urlset> element with one <url> per page. Inside it, <loc> holds the full address and is the only required child. <lastmod> is the date the page last changed, <changefreq> a guess at how often it will change, and <priority> a value from 0.0 to 1.0 that ranks the page against the site’s own other pages, never against anyone else’s. The protocol also accepts a plain text file with one address per line, or an RSS or Atom feed.

The protocol sets hard edges. One file holds at most 50,000 addresses and 50 MB uncompressed, may be gzipped, and must be UTF-8 with its values entity-escaped; each address stays under 2,048 characters. A larger site splits its list and publishes a sitemap index, a second kind of file that names up to 50,000 sitemaps on the same site. A sitemap may also list only addresses at or below its own location: a file at /catalog/sitemap.xml covers the /catalog/ pages and stops at that folder’s edge.

A search engine learns where the file lives in one of two ways. A Sitemap: line in robots.txt gives its full address, independently of any user-agent group, or the owner submits it through the engine’s own tools, such as Google Search Console.

The same word also names a page of links for human visitors; when the difference matters, people say “XML sitemap” for the one written for programs.

Why it matters

Google’s documentation is candid about the limits. A site whose pages are properly linked can usually be discovered without a sitemap, and Google draws the line for “small” at about 500 pages that should appear in results. It also says a sitemap “doesn’t guarantee that all the items in your sitemap will be crawled and indexed”, and that submitting one “is merely a hint”. The cases where it helps most are large sites, new sites with few links from elsewhere, and sites heavy with video, images or news.

Almost every site generator writes one for free, so the real decision is what the file claims, and every entry is a statement an engine can check against the page itself.

Which addresses go in? Google asks for the canonical address of each page. A list padded with redirects, duplicates or pages marked noindex asks for exactly what the rest of the site says to leave alone.

Which fields? Google says it ignores <priority> and <changefreq>. It uses <lastmod> only “if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate”, and the protocol says the date “must be set to the date the linked page was last modified, not when the sitemap is generated”. Google counts changes to a page’s main content, structured data or links as significant, and a new copyright year as trivial.

That is where the cost sits. A date from the file system is free and usually wrong, because deploys, exports and fresh checkouts touch files whose words stayed the same. A date from version history is honest, but the build then needs that history present and read in the right order. A date typed by hand is precise until someone forgets to change it.

What it looks like in practice

This site’s robots.txt names one address, https://engineer.company/sitemap.xml, and that file is a sitemap index naming three sitemaps, one each for English, Danish and Ukrainian. Each entry carries the address and a date, plus links to the same page in the other two languages. <priority> and <changefreq> are left out, since Google says it ignores both and the protocol makes neither of them required. When this article was written, the entry for the page on AI readiness read like this, its language links set aside:

<url>
  <loc>https://engineer.company/ai-ready/</loc>
  <lastmod>2026-10-15T14:25:16+02:00</lastmod>
</url>

The list is selective on purpose. The company’s focused satellite sites republish work from here with a canonical pointing back, and the same template keeps those copies out of each satellite’s sitemap: an entry asks an engine to index that exact address, and asking for a copy while naming another page as the original would contradict itself. Each host announces only its own pages.

The dates have a history of their own. At one point the sitemap told search engines that 172 of its 176 pages had changed at the same instant. The cause was ordinary: the date came from each file’s modification time, and the content export rewrites every file on every run. The fix moved the source to version history, with a fallback chain that tries an explicit field, then the commit history, then the file. It held one trap worth knowing: the literal field name has to appear in that list, or the generator never reads it. Afterwards, a re-sync following a month of edits touched eight files of 186 instead of all of them, and the sitemap says something true.

That date now does work beyond the sitemap. As the page on being found by AI agents describes, every deploy announces what changed through IndexNow, a protocol that Bing and several other engines take part in. The deploy script decides what changed by comparing each page’s lastmod with what it last sent, so a deploy that edits one page announces one page. Google is not on IndexNow’s own list of participating engines, so it learns of the change when it next reads the sitemap. A guard in the build keeps the dates honest: it reads every sitemap date against the page’s history and refuses to publish when one is wrong. For people rather than programs there is also a readable index of every page.

Where it goes wrong

Every page claims to have changed today. The sitemap carries one date, or one per deploy, across hundreds of pages, and an engine reading it learns nothing about which work is new. The cause is a date taken from the file system or the build clock. Google uses <lastmod> only when it is consistently and verifiably accurate, which a shared timestamp visibly is not. Take the date from the content’s own history, and compare a sample of entries against it after any change to the build.

The list asks for pages the site wants left out. Redirects, duplicate addresses carrying tracking parameters, pages marked noindex, staging hosts and copies whose canonical points elsewhere all appear in sitemaps written by a generator that lists every file it built. Each one sends a crawler to an address the page itself then disowns. List only the canonical address of each page meant for results, and let the sitemap and the canonical tag answer from one rule, as this site’s templates do.

Entries fall outside what the file may cover. A sitemap at /blog/sitemap.xml lists shop pages, or writes relative paths such as /page.html. The protocol limits a file to addresses at or below its own folder, and Google crawls addresses exactly as written. Publish the sitemap at the root, write every <loc> as a complete https:// address on the same host, and switch to a sitemap index before the list nears 50,000 entries or 50 MB.

Pages listed for months stay out of the index. The owner reads the sitemap as an instruction and waits. Google calls it a hint: a listed address can still be skipped or folded into a duplicate for reasons outside the file. Look at the page rather than the list: whether other pages link to it, whether its canonical names itself, and whether robots.txt or a noindex rule keeps a crawler away.

Further reading

Glossary