{"canonical_url":"https://engineer.company/glossary/unicode/","content_markdown":"# Unicode\n\n## What it is\n\nUnicode is the standard that gives every letter, digit, punctuation mark and\nsymbol of the world's writing systems a number of its own. Its maintainers call\nit the universal character encoding: built so that text in the languages and\ntechnical fields of the modern world can be exchanged, processed and displayed\nanywhere, and also able to represent older scripts and historical writing. One\nrepertoire holds the Danish å, the Ukrainian Ґ, the euro sign and the letters of\nscripts nobody has written for centuries, so a single document, or a single\ndatabase column, can contain all of them.\n\nStrictly speaking, a Unicode release includes several coordinated parts: its\ncore specification, charts of characters, the annexes and a Character Database,\nthe machine-readable files that record the properties of every character. The\nUnicode Technical Committee maintains it continuously, the Unicode Consortium\npublishes it, and version 18.0 was announced on 16 September 2026.\n\nThe number a character receives is its code point. The codespace runs from 0 to\n10FFFF in hexadecimal, 1,114,112 code points in all, and a code point is written\nas U+ followed by at least four hex digits: å is U+00E5, Ґ is U+0490 and the\neuro sign is U+20AC. The Consortium's stability policy promises that an encoded\ncharacter keeps its place for good: it is never moved or taken out, and its name\nnever changes, so text encoded today means the same thing in every later\nversion.\n\nTwo distinctions carry most of the standard. The first is between a character\nand a glyph. Unicode encodes the character, the abstract letter, and leaves its\nshape to the font: the core specification states that glyph shapes are the\nresponsibility of font vendors and not part of the standard. The same code point\ncan look different in two typefaces, and a typeface with no shape for it cannot\ndraw it at all. The second is between a code point and the bytes that store it.\nThe three Unicode encoding forms are UTF-8, UTF-16 and UTF-32, which write each\ncode point as one or more units of 8, 16 or 32 bits, and the specification calls\nit a trap to try to distinguish \"UTF-8 versus Unicode\": all three are equally\nvalid ways to implement the same characters.\n\n## Why it matters\n\nBefore Unicode, each script had its own code pages, and a byte meant nothing\nuntil you knew which page was in use. In the Western European code page\nwindows-1252 the byte E5 is å; in the Cyrillic windows-1251 the same byte is the\nletter е. Neither page could hold the other's letters: windows-1251 has Ґ and no\nå, æ or ø, and windows-1252 has the reverse. A product in Danish and Ukrainian\nneeded two encodings and a reliable record of which text used which, and a\nsentence that quoted both languages fit in neither encoding.\n\nUnicode removed that bookkeeping. Every language a product publishes, and every\nname a customer types into a form, fits one character set, and the same number\nmeans the same character on every system that reads it. That is what lets a\ndatabase, an API and a browser pass text between them without first agreeing on\na code page.\n\nThe standard also settles questions that every program would otherwise settle\nits own way. It unifies characters across languages: the comma is one character\nwhether it separates thousands, as in English, or decimals, as in French, so the\nmeaning comes from the locale and not from the code point. It does not unify\nacross scripts: the Latin o, the Cyrillic о and the Greek ο look the same in\nmost typefaces and are three characters, because they belong to three scripts\nand older encodings already kept them apart. And it records which sequences are\nequivalent. Å can be stored as one precomposed character or as A followed by a\ncombining ring, and the standard defines the two as canonically equivalent, to\nbe displayed and interpreted the same way, with normalization forms that turn\neither into one agreed spelling.\n\nThat generality has a price, and the programmer pays it. A character is not a\nbyte, not always one unit of a string and not always what a reader counts as one\nletter. Code that treats the three as the same passes its tests on English data\nand fails on the names, addresses and messages of the people it was written for.\n\n## What it looks like in practice\n\nJavaScript makes the layers visible. A string is a sequence of UTF-16 code\nunits, and spreading it into an array splits it into code points:\n\n```js\nconst hex = (n) =\u003e n.toString(16).toUpperCase().padStart(4, \"0\");\nfor (const text of [\"a\", \"å\", \"Ґ\", \"€\", \"𐌂\"]) {\n  const points = [...text].map((c) =\u003e \"U+\" + hex(c.codePointAt(0)));\n  console.log(text, points.join(\" \"), text.length, [...text].length);\n}\n```\n\nRun in Node.js 25, it prints each character, its code point, its length in\nUTF-16 units and its length in code points:\n\n```text\na U+0061 1 1\nå U+00E5 1 1\nҐ U+0490 1 1\n€ U+20AC 1 1\n𐌂 U+10302 2 1\n```\n\nThe last line is an Old Italic letter. Its code point lies beyond U+FFFF, so\nUTF-16 stores it as two units, a surrogate pair, and `length` reports 2 for what\nis one character.\n\nEquivalence shows up the same way. Here are three strings that all display as Å:\n\n```js\nconst forms = [\"\\u00C5\", \"A\\u030A\", \"\\u212B\"];\nconsole.log(\"form length equal NFC\");\nfor (const [i, s] of forms.entries()) {\n  console.log(i + 1, s.length, s === forms[0], s.normalize(\"NFC\") === forms[0]);\n}\n```\n\n```text\nform length equal NFC\n1 1 true true\n2 2 false true\n3 1 false true\n```\n\nThe first is the precomposed letter, U+00C5. The second is a plain A followed by\nU+030A, the combining ring above. The third is U+212B, the angstrom sign, which\nthe Unicode Character Database maps to U+00C5. Compared as they are, they are\nthree different strings; after normalization to NFC, all three are the same one.\nNothing on screen tells them apart, so a search, a unique index or a login that\ncompares the raw strings sees three different names.\n\nThis site depends on code points in a less visible place. The note about [the\nunderline that crossed every letter](/notes/safari-underline-font-subset/)\ndescribes the site's Latin and Cyrillic Ubuntu body fonts as six cuts, three\nweights of Latin and three of Cyrillic, so that a Ukrainian page is set in the\nsame typeface as an English one. Each cut declares the range of characters it\ncovers. Those ranges guide font selection for the characters on the page; an\nexplicit preload can also request a file before normal font discovery. The\nCyrillic declaration states its range in code points:\n\n```text\nunicode-range: U+0400-045F, U+0490-0491;\n```\n\nThe first range runs from Ѐ to џ and holds the Ukrainian Є, І and Ї along with\nthe rest of the basic Cyrillic alphabet. The second holds only Ґ and ґ, which\nsit further along in the Cyrillic block. The ranges declare which face is eligible for a character,\nbut the font must also contain its glyph. They do not guarantee that a page\nfetches files from only one script. English and Danish text uses the Latin faces, but the Ukrainian\nlanguage label can need a Cyrillic face, and the site explicitly preloads the\nlight Cyrillic body cut on non-Cyrillic pages for screen use.\n\n## Where it goes wrong\n\n**Length is counted in code units.** JavaScript's `length` counts UTF-16 units,\nso 𐌂 counts as two, and an A with a combining ring counts as two where a reader\nsees one letter. A limit enforced in units rejects text that fits, and a cut at\na fixed position can split a character: `\"A\\u030A\".slice(0, 1)` keeps the A and\ndrops the ring, and cutting 𐌂 in half leaves a lone surrogate that the UTF-8\nencoder replaces with U+FFFD, the replacement character. Decide what each limit\ncounts: bytes where storage sets it, characters as a reader sees them where a\nperson does. `Intl.Segmenter` counts the latter, and for Århus written with a\ncombining ring it returns 5 where `length` returns 6. Cut text only at the\nboundaries it reports.\n\n**Equal text is compared unit by unit.** The three forms of Å above are three\ndifferent strings to `===`, to a hash and to a unique index, so a customer can\nregister the same name twice, or fail to find a document by the title they\ntyped. Normalize text to NFC where it enters the system, store it normalized,\nand compare normalized forms. The stability policy guarantees that a string\nnormalized under one version stays normalized under every later one, provided\nthe string contains only characters that version knows, so the normalizing code\nhas to be as current as the text.\n\n**Look-alike characters pass as the same.** The Latin o, the Cyrillic о and the\nGreek ο are different code points that most typefaces draw alike, so two account\nnames or two product codes can look identical and differ in a single character.\nUnicode keeps them apart on purpose, and its standard on security mechanisms,\nUTS #39, sets out how to reduce that kind of spoofing. Where a name identifies a\nperson or a payee, allow one script per name, or test it against the standard's\nsecurity data before accepting it.\n\n**Bytes are read in the wrong encoding.** Text stored as UTF-8 and read as\nwindows-1252 turns blåbær into blÃ¥bÃ¦r, because each two-byte letter is read as\ntwo one-byte ones. The word Unicode does not say how the bytes were written;\nonly the encoding form does. Declare it wherever text crosses a boundary, in the\nHTTP header, in the page, on the database connection and in the file format, and\ntreat a file that arrives without one as a question for a person rather than a\nguess for the code.\n\n**A missing glyph is blamed on the text.** When a letter shows up as an empty\nbox, the usual cause is a font with no shape for that code point, and the text\nitself is intact. Because Unicode leaves shapes to fonts, every typeface a\nproduct uses has to cover every character its languages need, Є, І, Ї and Ґ as\nwell as А to я, or the browser draws them in whatever fallback it finds. The\nnote above gives this as the reason the site serves Cyrillic at all: without it,\na Ukrainian page would fall back to whatever the system offers. Test each\nlanguage with its whole alphabet rather than a sample sentence.\n\n**Names are checked against A to Z.** A pattern such as `^[A-Za-z' -]+$` accepts\nO'Neill and rejects Søren Ågård, Zoë and Ґалаґан, the very names a Danish or\nUkrainian product exists to serve. Test for letters by their Unicode properties\ninstead: in JavaScript, `/^[\\p{L}\\p{M}' -]+$/u` accepts all four. Those\nproperties come from the copy of the Unicode Character Database inside the\nruntime. For example, the Node.js 25.2.1 runtime used for the examples here\nreports Unicode 16.0; that is a specific build, not a guarantee for every 25.x\nrelease. Letters added in later Unicode versions fail the test until the\nruntime's character data is upgraded. Validate names loosely, and let a person\nrather than a pattern decide the hard cases.\n\n## Further reading\n\n- [About the Unicode Standard](https://www.unicode.org/standard/standard.html)\n  — the Consortium's own description of the standard, what makes up a version,\n  and where the latest one is published.\n- [The Unicode Standard, Chapter 2: General Structure](https://www.unicode.org/versions/Unicode18.0.0/core-spec/chapter-2/)\n  — the ten design principles, the codespace, the three encoding forms and the\n  rules for equivalent sequences.\n- [Unicode Character Encoding Stability Policies](https://www.unicode.org/policies/stability_policy.html)\n  — what can never change once a character is encoded, from its code point and\n  name to its normalization.\n- [Unicode 18.0.0](https://www.unicode.org/versions/Unicode18.0.0/) — what the\n  current version adds, and the technical standards versioned with it, among\n  them collation and security.\n- [Node.js: process.versions](https://nodejs.org/api/process.html#processversions)\n  — the runtime's component-version report, used here to pin the Unicode-data\n  example to one Node.js build rather than an entire release line.\n- [CSS Fonts: character ranges](https://www.w3.org/TR/css-fonts-4/#descdef-font-face-unicode-range)\n  — how declared character ranges and the font’s actual glyph coverage together\n  determine which characters a face can render.\n\n\u003chttps://engineer.company/glossary/unicode/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"How Unicode numbers every character, why one letter can be stored three ways, and six mistakes that break names, lengths and comparisons in real text.","formats":{"gemtext":"https://engineer.company/glossary/unicode/index.gmi","gopher":"https://engineer.company/glossary/unicode/gophermap","html":"https://engineer.company/glossary/unicode/","json":"https://engineer.company/glossary/unicode/index.json","markdown":"https://engineer.company/glossary/unicode/index.md","pdf":"https://engineer.company/glossary/unicode/index.pdf"},"kind":"page","language":"en","title":"What is Unicode?","url":"https://engineer.company/glossary/unicode/","version":1}