# UTF-8

## What it is

UTF-8 is the encoding that turns Unicode text into bytes. The Unicode Standard
defines it as the encoding form that stores each Unicode scalar value, which is
any code point except the surrogates set aside for UTF-16, as a sequence of one
to four bytes. It is one of the standard's three encoding forms, beside UTF-16
and UTF-32, and the one the web's own standards require. As a label, in an HTML
page or an HTTP header, it is usually written in lower case, utf-8, and
sometimes without the hyphen, utf8; the Encoding Standard accepts both and
ignores case in labels.

Ken Thompson designed UTF-8 in September 1992 for the Plan 9 operating system,
working to criteria that Rob Pike had laid down, and on its way through
standardisation the encoding was called FSS-UTF and then UTF-2 before it got its
present name. RFC 3629, the IETF's definition from 2003, limits it to the code
points U+0000 to U+10FFFF, the same codespace Unicode has. When the RFC was
written, the description of UTF-8 in ISO/IEC 10646 still allowed sequences of up
to six bytes; under the RFC and the Unicode Standard, no character takes more
than four.

The leading bits of every byte say what kind of byte it is. A byte whose top bit
is 0 is a character on its own, one of the 128 characters of ASCII. A byte that
begins with 110, 1110 or 11110 starts a sequence of two, three or four bytes,
and each byte that follows begins with 10 and carries six more bits of the code
point. The whole format fits in four rows, with x standing for the bits of the
code point:

```text
U+0000..U+007F     0xxxxxxx
U+0080..U+07FF     110xxxxx 10xxxxxx
U+0800..U+FFFF     1110xxxx 10xxxxxx 10xxxxxx
U+10000..U+10FFFF  11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
```

The three-byte row leaves out the surrogates, U+D800 to U+DFFF: they are not
characters, and any byte sequence that would encode one is ill-formed.

Four properties follow from that shape, and the Unicode Standard names each of
them. UTF-8 is transparent to ASCII: the bytes 00 to 7F mean what they mean in
ASCII and never occur inside another character. It is self-synchronising: a lead
byte is easy to tell from a continuation byte. Sorting UTF-8 strings byte by
byte puts them in the same order as sorting them by code point. And it has no
byte order, so the question of big-endian or little-endian order that UTF-16 and
UTF-32 have to answer never arises.

## Why it matters

The web has settled on it. The WHATWG Encoding Standard, the specification
browsers implement for reading and writing text, identifies UTF-8 as the
preferred way to exchange Unicode text and requires it for new protocols and
formats, and for existing formats deployed in new contexts. It tells authors to
use UTF-8 and to identify it by the label utf-8. HTML follows suit: the encoding
of a document must be UTF-8 whether or not the document declares it, and
authoring tools must default to UTF-8 for new documents. A product that
publishes pages, sends JSON or accepts form posts is already expected to speak
it.

The design explains why that settlement held. Because the ASCII bytes never
appear inside another character, a program written for ASCII keeps working on
UTF-8 text: a parser that splits on commas, slashes or line breaks finds them in
a Danish or Ukrainian sentence exactly where they are, and never in the middle
of an å or a Ґ. Because a program dropped at any position finds the start of a
character by searching at most four bytes backward, and usually fewer, a damaged
byte costs one character instead of the rest of the file, and a search for one
character never matches part of another. Because the binary order is the code
point order, a database or a program can sort UTF-8 bytes without decoding them.
That order is a technical one, though: it puts å before æ and ø, where a Danish
dictionary ends the alphabet with æ, ø and å.

Size is the trade. The Unicode Standard calls UTF-8 reasonably compact: for
ASCII syntax and Western languages it takes much less space than UTF-16, while
text in Asian scripts such as Hindi, Thai, Chinese, Japanese and Korean comes
out significantly bigger. Danish text costs one byte per letter apart from æ, ø
and å, which take two. Ukrainian takes two bytes for every Cyrillic letter and
one for the spaces, digits and punctuation between them, so a field measured in
bytes holds about half as many Ukrainian letters as English ones.

## What it looks like in practice

The bytes are easy to see. Node.js, like a browser, has the Encoding Standard's
`TextEncoder`, which writes a string as UTF-8:

```js
const hex = (bytes) =>
  [...bytes].map((b) => b.toString(16).padStart(2, "0")).join(" ");
const encoder = new TextEncoder();
for (const text of ["A", "å", "Ґ", "€", "𐌂"]) {
  console.log(text, hex(encoder.encode(text)).toUpperCase());
}
```

Run in Node.js 25, it prints each character and its bytes in hexadecimal:

```text
A 41
å C3 A5
Ґ D2 90
€ E2 82 AC
𐌂 F0 90 8C 82
```

The letter A is ASCII and keeps its single byte. The Danish å is U+00E5, which
needs eight bits, one more than the seven a single byte offers, so it takes the
two-byte pattern. Its eleven-bit value, 00011100101, is split into 00011
and 100101; the lead byte becomes 110 followed by 00011, which is C3, and the
continuation byte becomes 10 followed by 100101, which is A5. The Ukrainian Ґ,
U+0490, takes two bytes the same way, the euro sign three, and the Old Italic
letter 𐌂, beyond U+FFFF, four. The first byte of each sequence announces its
length: C3 and D2 begin with 110, E2 with 1110 and F0 with 11110.

This site writes every page in UTF-8 and says so where HTML asks for it. The
first element in the head of every page is `<meta charset="utf-8">`, with the
label utf-8 that the Encoding Standard prescribes, and well inside the
first 1,024 bytes, where HTML requires the declaration to sit. The
[credits page](/credits/) has a section of outside checks, none of them ours,
each running a live test instead of taking our word for anything. One of them
is the W3C Internationalization Checker, which reads the language and encoding
declarations that make every language version of this site legible to the right
reader.

## Where it goes wrong

**A byte limit cuts a character in half.** A column, a message field or a buffer
measured in bytes can end in the middle of a sequence. The name Ærø is three
characters and five bytes, and a limit of four keeps Æ, r and the first byte of
ø:

```js
const bytes = new TextEncoder().encode("Ærø");
const cut = new TextDecoder().decode(bytes.slice(0, 4));
console.log(bytes.length, [...cut].map((c) => c.codePointAt(0).toString(16)));
console.log(new TextEncoder().encodeInto("Ærø", new Uint8Array(4)));
```

```text
5 [ 'c6', '72', 'fffd' ]
{ read: 2, written: 3 }
```

The orphaned lead byte decodes as U+FFFD, the replacement character, so the
stored name ends in a placeholder where the ø was. Cut on a character boundary
instead. `encodeInto`, on the last line, does that by itself: given four bytes
of room, it writes only the three bytes of Æ and r, and reports in `read` how
much of the string it took, counted in UTF-16 units. Code that cuts bytes it
already holds can step back over the continuation bytes, the ones that begin
with 10, to the lead byte, and cut there.

**A byte order mark starts the file.** UTF-8 has no byte order, so it needs no
mark, but a converter that turns UTF-16 with a byte order mark into UTF-8 turns
the mark into the three bytes EF BB BF. The Unicode Standard does not require
them, says their presence does not affect conformance, and does not recommend
them as a signature for UTF-8. Readers disagree about them. Here the file
bom.json holds those three bytes followed by `{"name":"Ærø"}`:

```js
import { readFileSync } from "node:fs";

const text = readFileSync("bom.json", "utf8");
console.log(text.codePointAt(0).toString(16));
try {
  JSON.parse(text);
} catch (error) {
  console.log(error.name);
}
console.log(JSON.parse(new TextDecoder().decode(readFileSync("bom.json"))));
```

```text
feff
SyntaxError
{ name: 'Ærø' }
```

Node's `readFileSync` keeps the mark as the character U+FEFF, and `JSON.parse`
rejects the text because of it. `TextDecoder` follows the Encoding Standard,
removes the mark, and the same bytes parse cleanly. In a browser the mark goes
further: the Encoding Standard ranks it above anything else, the HTTP
`Content-Type` header included. Write UTF-8 without the mark, and where files
arrive from other systems, decode them with a decoder that strips it, or strip a
leading U+FEFF before parsing.

**A check reads the bytes before they are decoded.** RFC 3629 describes a parser
that checks its input in encoded form and refuses the bytes 2F 2E 2E 2F, the
path step "/../", yet lets through 2F C0 AE 2E 2F, in which C0 AE is an illegal
two-byte spelling of the full stop that the parser goes on to read as one. The
RFC records that a virus which spread widely among web servers in 2001 used
exactly this trick. Since Unicode 3.1 such non-shortest forms are ill-formed,
and the bytes C0, C1 and F5 to FF never occur in valid UTF-8. A conforming
decoder, `TextDecoder` among them, turns those five bytes into a slash, two
U+FFFD and "./", so no "/../" is left to slip past. The Unicode Standard is just
as strict about what follows an error: a decoder that swallows the valid bytes
after a bad lead byte is non-conformant and open to security exploits, and
`TextDecoder` turns the bytes C2 41 42 into U+FFFD followed by AB. Decode first
with the platform's own decoder, then check the decoded text, and never write a
UTF-8 decoder, or a byte pattern that stands in for one, by hand.

**Replacement hides the damage.** By default a decoder replaces every bad
sequence with U+FFFD and carries on. That is right for showing a page and wrong
for importing data: the text reads almost normally, the stored copy has lost the
original bytes, and the sender never learns where the file was damaged. The
Encoding Standard names two error modes for decoding, "replacement" and "fatal",
and for identifiers and byte sequences inside a format or protocol it lets other
standards pick a decoder that fails instead of replacing. In JavaScript,
`new TextDecoder("utf-8", { fatal: true })` throws a `TypeError` for the cut
name above instead of returning it. Use the fatal mode wherever text is stored,
signed or compared, reject the input with a message the sender can act on, and
keep replacement for display.

**The declaration is late, missing or wrong.** HTML requires the element that
declares the encoding to be complete within the first 1,024 bytes of the
document. It requires a declaration even when every character is ASCII, because
the browser needs an encoding for what people type into forms and for URLs that
scripts build. A label that looks harmless can also be the wrong one: in the
Encoding Standard, "ascii", "latin1" and "iso-8859-1" are all labels of
windows-1252, so a page declared as ASCII is read as windows-1252. Put
`<meta charset="utf-8">` first in the head, send
`Content-Type: text/html; charset=utf-8` with every page, and treat any other
label in either place as a defect.

## Further reading

- [The Unicode Standard, Chapter 3: Conformance](https://www.unicode.org/versions/Unicode18.0.0/core-spec/chapter-3/#G27817)
  — the definition of UTF-8, the table of well-formed byte sequences, the byte
  order mark, and how a converter must treat ill-formed input.
- [The Unicode Standard, Chapter 2: General Structure](https://www.unicode.org/versions/Unicode18.0.0/core-spec/chapter-2/)
  — why UTF-8 is transparent to ASCII, self-synchronising and free of byte
  order, and how its size compares with UTF-16.
- [RFC 3629: UTF-8, a transformation format of ISO 10646](https://www.rfc-editor.org/rfc/rfc3629)
  — the IETF definition, its history from Plan 9, and the security
  considerations behind the overlong path attack.
- [Encoding Standard](https://encoding.spec.whatwg.org/) — the WHATWG
  specification browsers follow: why UTF-8 is required, how decoding treats a
  byte order mark and errors, and the `TextEncoder` and `TextDecoder`
  interfaces.
- [HTML: Specifying the document's character encoding](https://html.spec.whatwg.org/multipage/semantics.html#charset)
  — what a declaration must say, where it must appear, and why a page written
  only in ASCII needs one.

<https://engineer.company/glossary/utf-8/>