{"canonical_url":"https://engineer.company/glossary/metadata/","content_markdown":"# Metadata\n\n## What it is\n\nMetadata is data whose subject is other data: the title, date, language, author,\nformat and size that travel with a document, so that people and programs can\nsort it, trust it and act on it before reading it. Older standards hyphenate it\nas meta-data, and day to day it goes by narrower names: header fields in HTTP,\nproperties in an office file, tags in a photo, front matter at the top of a\nMarkdown page.\n\nA common split has three kinds. Descriptive metadata says what a thing is about:\ntitle, creator, subject, summary. Structural metadata says how its parts fit\ntogether: chapters, pages, reading order. Administrative metadata says where it\ncame from and what may be done with it: when it was created and changed, by\nwhom, under which licence, in which format. The Dublin Core, fifteen elements\nthat became the standard ISO 15836 in 2003, is the best-known shared list of\ndescriptive fields, and its title, creator, date, language and rights reappear\nunder other names in almost every format since.\n\nA web page carries metadata in layers. The HTTP response comes first. RFC 9110\nsays a header field \"might modify or extend message semantics, describe the\nsender, define the content, or provide additional context\", and `Content-Type`,\n`Content-Language` and `Last-Modified` describe the page before a byte of it\narrives. Inside the HTML sit the `\u003ctitle\u003e`, the `lang` attribute and `\u003cmeta\u003e`\nelements whose names the HTML standard defines, such as `description`, `author`\nand `generator`. Above those come vocabularies that other parties read: Open\nGraph properties for link previews, and structured data for search engines and\nassistants.\n\nSo what separates metadata from data? The reader, more than the bytes. A\ncatalogue record is metadata about a book and the whole dataset to the\ncatalogue; a photo's capture time is a detail to the person looking at it and\nthe raw material for someone mapping where photos are taken. Metadata versus\nstructured data is the narrower question, and the answer is that structured data\nis metadata written in a shared vocabulary, so that a program can read a field\nwithout guessing what it means.\n\n## Why it matters\n\nEach field is a small promise, and the decision that matters is where each one\ngets its value: a person typing it, the version history, a database column, or\nwhatever the filesystem happens to report. Programs act on those promises\nwithout checking them. That is the point of metadata, and its risk.\n\nCaches and crawlers spend their effort by it. A browser holding a copy of a page\nsends back the `Last-Modified` date it was given, and when nothing has changed\nit receives a short `304 Not Modified` instead of the page again. RFC 9110 asks\na server to send that date for any representation whose last change \"can be\nreasonably and consistently determined\", and the condition cuts both ways: where\nno consistent date exists, the standard does not ask for one. A sitemap's\nmodification date does the same job for a crawler choosing what to fetch again.\n\nPeople and assistive technology depend on it too. The `lang` attribute tells a\nscreen reader which pronunciation rules to apply, and the title is what a tab, a\nbookmark and a search result show. Rights travel the same way: a package manager\nknows a dependency's licence only because the dependency declared it, so a\nlicence review built on that list is exactly as good as the declarations\nunderneath.\n\nEvery field has to stay true, so a page with forty fields has forty chances to\nbe wrong. Some fields are read by nobody: the HTML standard notes that many\nsearch engines do not consider the `keywords` name, because it was used to spam\nresults. And metadata leaks, since a photo's position or a document's revision\nhistory can say more than the content does. Fewer fields, each filled from a\nsource that cannot drift, usually beat many.\n\n## What it looks like in practice\n\nMetadata turns up in three projects on this site, each time in a different kind\nof file: a sitemap, a set of PDFs and a dependency graph.\n\nOne site's sitemap took its last-changed dates from the wrong metadata, and\n[172 of its 176 pages ended up sharing one instant](/portfolio/fixed-a-sitemap-with-one-shared-timestamp/).\nThose dates were file modification times, metadata about the bytes rather than\nthe content, and each run of the content export wrote all of them anew. The fix\nreads a fallback chain, an explicit field first, then the commit history, then\nthe file, and it records the trap inside it: the\nliteral field name has to appear in that list, or the generator never reads the\nfield at all. The records behind the pages now keep the day each entry was\nwritten and revised apart from the years the work itself took place, and a date\nnobody knows stays empty, on the rule that \"a precise wrong number is worse than\nan absent one\". At the recorded implementation stage, a re-sync after a month of edits touched 8 files of 186.\n\nAt the recorded validation stage,\n[nine generated PDF documents measured against the accessibility standard](/portfolio/reached-pdf-ua-1-conformance-at-106-of-106/)\npassed all 106 configured accessibility-profile rules. The work included a declared document\nlanguage and a real title in the metadata, alongside tagged structure, alternative text and an\nexplicit reading order. In that recorded run, the same documents failed exactly one rule of\n146 in the archival profile. The page calls that a near miss with the failing rule named,\nbecause 145 passes is not conformance.\n\nAnd when\n[the licences of 644 Rust crates were checked on every build](/portfolio/know-the-licence-of-everything-you-ship/),\nthe stated limit of the audit was a limit of metadata. It reads each crate's\ndeclared licence, declarations can be wrong or incomplete, and the check proves\nnothing about a crate that misdeclares itself. It was made to fail on purpose\ntwice, which shows the check works; nothing a check does can show that the\ndeclarations it reads are honest.\n\n## Where it goes wrong\n\nThe faults below share one trait: every check that is not looking for them\npasses.\n\n**Every page reports a change after every deploy.** A web server that sends a\nstatic file's modification time as `Last-Modified`, or a generator that dates a\nsitemap the same way, reports the moment the file was copied, and a checkout, an\nexport or a container build copies all of them. Caches revalidate everything and\ncrawlers learn nothing. Take dates from the version history or from a field a\nperson wrote, and let a date nobody knows stay absent. In Hugo the sources are a\nlist read in order, and the field's own name has to be in it:\n\n```toml\nenableGitInfo = true\n\n[frontmatter]\ndate = [\"date\", \":git\", \":fileModTime\"]\nlastmod = [\"lastmod\", \":git\", \":fileModTime\"]\n```\n\n**A title or a link preview shows broken characters.** The character encoding is\nmetadata about every other byte, and a parser that meets text before the\ndeclaration has to guess. The HTML standard requires the declaring element to\nfit completely within the first 1,024 bytes, and putting it first in `\u003chead\u003e`\nleaves nothing to guess. On this site, Bing's URL Inspection once read the title\nand both Open Graph strings as Latin-1, because an em dash in the title arrived\nbefore the charset. The HTTP header named UTF-8 as well, but a tool reading the\nbytes without the header has only the page to go on. The charset is now the\nfirst element, and a check confirms that order on every built page.\n\n**The same fact says different things in different places.** A page's title\nlives in `\u003ctitle\u003e`, in `og:title`, in its structured data and, for a PDF, in the\ndocument properties; its language lives in `lang`, in `Content-Language` and in\n`og:locale`. Typed separately, they drift apart, and each consumer believes a\ndifferent one. Compute each value once, in one template or function, and have\nevery place that prints it read that value.\n\n**A file says more than its author meant to publish.** Photos carry the camera,\nthe time and often the position; office documents carry author names, edit\nhistory and template paths; a `generator` tag names the software that built a\npage. Strip on export whatever nobody chose to publish, and treat every field as\ntext a stranger will read.\n\n## Further reading\n\n- [RFC 9110, section 6.3: Header Fields](https://www.rfc-editor.org/rfc/rfc9110#section-6.3)\n  — how HTTP carries metadata ahead of the content, and what a header field may\n  do.\n- [RFC 9110, section 8.8.2: Last-Modified](https://www.rfc-editor.org/rfc/rfc9110#section-8.8.2)\n  — when a server should send a modification date, and why one in the future is\n  replaced.\n- [HTML Standard: the meta element](https://html.spec.whatwg.org/multipage/semantics.html#the-meta-element)\n  — the standard metadata names and the rules for declaring a character\n  encoding.\n- [DCMI Metadata Terms](https://www.dublincore.org/specifications/dublin-core/dcmi-terms/)\n  — the fifteen Dublin Core elements and the vocabulary built around them.\n- [The Open Graph protocol](https://ogp.me/) — the four properties it requires\n  on every page, and the optional ones around them.\n\n\u003chttps://engineer.company/glossary/metadata/\u003e\n","date_modified":"2026-11-12T05:34:23Z","description":"Metadata in HTTP headers, HTML and file properties: why each field needs one trustworthy source, and how a confident wrong date misleads caches and crawlers.","formats":{"gemtext":"https://engineer.company/glossary/metadata/index.gmi","gopher":"https://engineer.company/glossary/metadata/gophermap","html":"https://engineer.company/glossary/metadata/","json":"https://engineer.company/glossary/metadata/index.json","markdown":"https://engineer.company/glossary/metadata/index.md","pdf":"https://engineer.company/glossary/metadata/index.pdf"},"kind":"page","language":"en","title":"What is metadata?","url":"https://engineer.company/glossary/metadata/","version":1}