{"canonical_url":"https://engineer.company/glossary/utf-8/","content_markdown":"# UTF-8\n\n## What it is\n\nUTF-8 is the encoding that turns Unicode text into bytes. The Unicode Standard\ndefines it as the encoding form that stores each Unicode scalar value, which is\nany code point except the surrogates set aside for UTF-16, as a sequence of one\nto four bytes. A user-perceived character can contain several scalar values\nand therefore need more bytes. It is one of the standard's three encoding forms, beside UTF-16\nand UTF-32, and the one the web's own standards require. As a label, in an HTML\npage or an HTTP header, it is usually written in lower case, utf-8, and\nsometimes without the hyphen, utf8; the Encoding Standard accepts both and\nignores case in labels.\n\nKen Thompson designed UTF-8 in September 1992 for the Plan 9 operating system,\nworking to criteria that Rob Pike had laid down, and on its way through\nstandardisation the encoding was called FSS-UTF and then UTF-2 before it got its\npresent name. RFC 3629, the IETF's definition from 2003, limits it to the code\npoints U+0000 to U+10FFFF, the same codespace Unicode has. When the RFC was\nwritten, the description of UTF-8 in ISO/IEC 10646 still allowed sequences of up\nto six bytes; under the RFC and the Unicode Standard, no Unicode scalar value takes more\nthan four.\n\nThe leading bits of every byte say what kind of byte it is. A byte whose top bit\nis 0 is a character on its own, one of the 128 characters of ASCII. A byte that\nbegins with 110, 1110 or 11110 starts a sequence of two, three or four bytes,\nand each byte that follows begins with 10 and carries six more bits of the code\npoint. The whole format fits in four rows, with x standing for the bits of the\ncode point:\n\n```text\nU+0000..U+007F     0xxxxxxx\nU+0080..U+07FF     110xxxxx 10xxxxxx\nU+0800..U+FFFF     1110xxxx 10xxxxxx 10xxxxxx\nU+10000..U+10FFFF  11110xxx 10xxxxxx 10xxxxxx 10xxxxxx\n```\n\nThe three-byte row leaves out the surrogates, U+D800 to U+DFFF: they are not\ncharacters, and any byte sequence that would encode one is ill-formed.\n\nFour properties follow from that shape, and the Unicode Standard names each of\nthem. UTF-8 is transparent to ASCII: the bytes 00 to 7F mean what they mean in\nASCII and never occur inside another character. It is self-synchronising: a lead\nbyte is easy to tell from a continuation byte. Sorting UTF-8 strings byte by\nbyte puts them in the same order as sorting them by code point. And it has no\nbyte order, so the question of big-endian or little-endian order that UTF-16 and\nUTF-32 have to answer never arises.\n\n## Why it matters\n\nThe web has settled on it. The WHATWG Encoding Standard, the specification\nbrowsers implement for reading and writing text, identifies UTF-8 as the\npreferred way to exchange Unicode text and requires it for new protocols and\nformats, and for existing formats deployed in new contexts. It tells authors to\nuse UTF-8 and to identify it by the label utf-8. HTML follows suit: the encoding\nof a document must be UTF-8 whether or not the document declares it, and\nauthoring tools must default to UTF-8 for new documents. A product that\npublishes pages, sends JSON or accepts form posts is already expected to speak\nit.\n\nThe design explains why that settlement held. Because the ASCII bytes never\nappear inside another character, a program written for ASCII keeps working on\nUTF-8 text: a parser that splits on commas, slashes or line breaks finds them in\na Danish or Ukrainian sentence exactly where they are, and never in the middle\nof an å or a Ґ. Because a program dropped at any position finds the start of an\nencoded scalar value by searching at most four bytes backward, and usually\nfewer, decoding can resume at a following well-formed sequence instead of losing\nthe rest of the file. That does not promise exactly one replacement character\nfor every damaged byte. A search for one complete encoded scalar value never\nmatches part of another. Because the binary order is the code point order, a\ndatabase or a program can sort UTF-8 bytes without decoding them. That order is\na technical one, though: it puts å before æ and ø, where a Danish dictionary\nends the alphabet with æ, ø and å.\n\nSize is the trade. The Unicode Standard calls UTF-8 reasonably compact: for\nASCII syntax and Western languages it takes much less space than UTF-16, while\ntext in Asian scripts such as Hindi, Thai, Chinese, Japanese and Korean comes\nout significantly bigger. In the basic Danish alphabet, letters take one byte\napart from æ, ø and å, which take two; other accented letters can need more.\nUkrainian takes two bytes for every Cyrillic letter and one for ASCII spaces,\ndigits and punctuation, but other punctuation can need more. A field measured\nin bytes therefore holds about half as many Ukrainian letters as English ones.\n\n## What it looks like in practice\n\nThe bytes are easy to see. Node.js, like a browser, has the Encoding Standard's\n`TextEncoder`, which writes a string as UTF-8:\n\n```js\nconst hex = (bytes) =\u003e\n  [...bytes].map((b) =\u003e b.toString(16).padStart(2, \"0\")).join(\" \");\nconst encoder = new TextEncoder();\nfor (const text of [\"A\", \"å\", \"Ґ\", \"€\", \"𐌂\"]) {\n  console.log(text, hex(encoder.encode(text)).toUpperCase());\n}\n```\n\nRun in Node.js 25, it prints each character and its bytes in hexadecimal:\n\n```text\nA 41\nå C3 A5\nҐ D2 90\n€ E2 82 AC\n𐌂 F0 90 8C 82\n```\n\nThe letter A is ASCII and keeps its single byte. The Danish å is U+00E5, which\nneeds eight bits, one more than the seven a single byte offers, so it takes the\ntwo-byte pattern. Its eleven-bit value, 00011100101, is split into 00011\nand 100101; the lead byte becomes 110 followed by 00011, which is C3, and the\ncontinuation byte becomes 10 followed by 100101, which is A5. The Ukrainian Ґ,\nU+0490, takes two bytes the same way, the euro sign three, and the Old Italic\nletter 𐌂, beyond U+FFFF, four. The first byte of each sequence announces its\nlength: C3 and D2 begin with 110, E2 with 1110 and F0 with 11110.\n\nThis site writes every page in UTF-8 and says so where HTML asks for it. The\nfirst element in the head of every page is `\u003cmeta charset=\"utf-8\"\u003e`, with the\nlabel utf-8 that the Encoding Standard prescribes, and well inside the\nfirst 1,024 bytes, where HTML requires the declaration to sit. The\n[credits page](/credits/) has a section of outside checks, none of them ours,\neach running a live test instead of taking our word for anything. One of them\nis the W3C Internationalization Checker, which reads the language and encoding\ndeclarations that make every language version of this site legible to the right\nreader.\n\n## Where it goes wrong\n\n**A byte limit cuts a character in half.** A column, a message field or a buffer\nmeasured in bytes can end in the middle of a sequence. The name Ærø is three\ncharacters and five bytes, and a limit of four keeps Æ, r and the first byte of\nø:\n\n```js\nconst bytes = new TextEncoder().encode(\"Ærø\");\nconst cut = new TextDecoder().decode(bytes.slice(0, 4));\nconsole.log(bytes.length, [...cut].map((c) =\u003e c.codePointAt(0).toString(16)));\nconsole.log(new TextEncoder().encodeInto(\"Ærø\", new Uint8Array(4)));\n```\n\n```text\n5 [ 'c6', '72', 'fffd' ]\n{ read: 2, written: 3 }\n```\n\nThe orphaned lead byte decodes as U+FFFD, the replacement character, so the\nstored name ends in a placeholder where the ø was. Cut on a Unicode scalar-value\nboundary instead. `encodeInto`, on the last line, does that by itself: given four bytes\nof room, it writes only the three bytes of Æ and r, and reports in `read` how\nmuch of the string it took, counted in UTF-16 units. Code that cuts bytes it\nalready holds can step back over the continuation bytes, the ones that begin\nwith 10, to the lead byte, and cut there.\nThat keeps the UTF-8 well-formed; preserving a whole user-perceived character\nalso requires respecting grapheme-cluster boundaries.\n\n**A byte order mark starts the file.** UTF-8 has no byte order, so it needs no\nmark, but a converter that turns UTF-16 with a byte order mark into UTF-8 turns\nthe mark into the three bytes EF BB BF. The Unicode Standard does not require\nthem, says their presence does not affect conformance, and does not recommend\nthem as a signature for UTF-8. Readers disagree about them. Here the file\nbom.json holds those three bytes followed by `{\"name\":\"Ærø\"}`:\n\n```js\nimport { readFileSync } from \"node:fs\";\n\nconst text = readFileSync(\"bom.json\", \"utf8\");\nconsole.log(text.codePointAt(0).toString(16));\ntry {\n  JSON.parse(text);\n} catch (error) {\n  console.log(error.name);\n}\nconsole.log(JSON.parse(new TextDecoder().decode(readFileSync(\"bom.json\"))));\n```\n\n```text\nfeff\nSyntaxError\n{ name: 'Ærø' }\n```\n\nNode's `readFileSync` keeps the mark as the character U+FEFF, and `JSON.parse`\nrejects the text because of it. `TextDecoder` follows the Encoding Standard,\nremoves the mark, and the same bytes parse cleanly. In a browser the mark goes\nfurther: the Encoding Standard ranks it above anything else, the HTTP\n`Content-Type` header included. Write UTF-8 without the mark, and where files\narrive from other systems, decode them with a decoder that strips it, or strip a\nleading U+FEFF before parsing.\n\n**A check reads the bytes before they are decoded.** RFC 3629 describes a parser\nthat checks its input in encoded form and refuses the bytes 2F 2E 2E 2F, the\npath step \"/../\", yet lets through 2F C0 AE 2E 2F, in which C0 AE is an illegal\ntwo-byte spelling of the full stop that the parser goes on to read as one. The\nRFC records that a virus which spread widely among web servers in 2001 used\nexactly this trick. Since Unicode 3.1 such non-shortest forms are ill-formed,\nand the bytes C0, C1 and F5 to FF never occur in valid UTF-8. A conforming\ndecoder, `TextDecoder` among them, turns those five bytes into a slash, two\nU+FFFD and \"./\", so no \"/../\" is left to slip past. The Unicode Standard is just\nas strict about what follows an error: a decoder that swallows the valid bytes\nafter a bad lead byte is non-conformant and open to security exploits, and\n`TextDecoder` turns the bytes C2 41 42 into U+FFFD followed by AB. Decode first\nwith the platform's own decoder, then check the decoded text, and never write a\nUTF-8 decoder, or a byte pattern that stands in for one, by hand.\n\n**Replacement hides the damage.** By default a decoder replaces every bad\nsequence with U+FFFD and carries on. That is right for showing a page and wrong\nfor importing data: the text reads almost normally, the stored copy has lost the\noriginal bytes, and the sender never learns where the file was damaged. The\nEncoding Standard names two error modes for decoding, \"replacement\" and \"fatal\",\nand for identifiers and byte sequences inside a format or protocol it lets other\nstandards pick a decoder that fails instead of replacing. In JavaScript,\n`new TextDecoder(\"utf-8\", { fatal: true })` throws a `TypeError` for the cut\nname above instead of returning it. Use the fatal mode wherever text is stored,\nsigned or compared, reject the input with a message the sender can act on, and\nkeep replacement for display.\n\n**The declaration is late, missing or wrong.** HTML requires the element that\ndeclares the encoding to be complete within the first 1,024 bytes of the\ndocument. It requires a declaration even when every character is ASCII, because\nthe browser needs an encoding for what people type into forms and for URLs that\nscripts build. A label that looks harmless can also be the wrong one: in the\nEncoding Standard, \"ascii\", \"latin1\" and \"iso-8859-1\" are all labels of\nwindows-1252, so a page declared as ASCII is read as windows-1252. Put\n`\u003cmeta charset=\"utf-8\"\u003e` first in the head, send\n`Content-Type: text/html; charset=utf-8` with every page, and treat any other\nlabel in either place as a defect.\n\n## Further reading\n\n- [The Unicode Standard, Chapter 3: Conformance](https://www.unicode.org/versions/Unicode18.0.0/core-spec/chapter-3/#G27817)\n  — the definition of UTF-8, the table of well-formed byte sequences, the byte\n  order mark, and how a converter must treat ill-formed input.\n- [The Unicode Standard, Chapter 2: General Structure](https://www.unicode.org/versions/Unicode18.0.0/core-spec/chapter-2/)\n  — why UTF-8 is transparent to ASCII, self-synchronising and free of byte\n  order, and how its size compares with UTF-16.\n- [RFC 3629: UTF-8, a transformation format of ISO 10646](https://www.rfc-editor.org/rfc/rfc3629)\n  — the IETF definition, its history from Plan 9, and the security\n  considerations behind the overlong path attack.\n- [Encoding Standard](https://encoding.spec.whatwg.org/) — the WHATWG\n  specification browsers follow: why UTF-8 is required, how decoding treats a\n  byte order mark and errors, and the `TextEncoder` and `TextDecoder`\n  interfaces.\n- [HTML: Specifying the document's character encoding](https://html.spec.whatwg.org/multipage/semantics.html#charset)\n  — what a declaration must say, where it must appear, and why a page written\n  only in ASCII needs one.\n\n\u003chttps://engineer.company/glossary/utf-8/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"How UTF-8 stores scalar values in one to four bytes, why the web requires it, and five ways cut strings, stray marks and bad bytes corrupt text.","formats":{"gemtext":"https://engineer.company/glossary/utf-8/index.gmi","gopher":"https://engineer.company/glossary/utf-8/gophermap","html":"https://engineer.company/glossary/utf-8/","json":"https://engineer.company/glossary/utf-8/index.json","markdown":"https://engineer.company/glossary/utf-8/index.md","pdf":"https://engineer.company/glossary/utf-8/index.pdf"},"kind":"page","language":"en","title":"What is UTF-8?","url":"https://engineer.company/glossary/utf-8/","version":1}