JSON

the text format most data travels in

What it is

JSON is a plain-text notation for nested data, written with braces, brackets, double quotes and commas so that a program in almost any language can read back exactly what another program wrote. The letters stand for JavaScript Object Notation, because the syntax was taken from the way JavaScript writes literal values, yet a JSON document belongs to no language and needs no JavaScript to read. Files usually end in .json, and over HTTP the content is labelled with the media type application/json.

The whole format fits on one page. Four primitive types (strings, numbers, the literals true and false, and null) and two structured ones make up the vocabulary: an object, which is an unordered set of name and value pairs, and an array, which is an ordered list. Anything can nest inside anything else. Dates, binary data, comments and references between parts of a document are all left to conventions outside the format.

Douglas Crockford first registered it with the IETF in RFC 4627, in July 2006. The current definition is RFC 8259 from December 2017, which agrees with Ecma’s standard ECMA-404 on the grammar and adds the rules that matter when two systems exchange it. Text that leaves a closed ecosystem must be UTF-8, and a sender must not add a byte order mark in front of it.

XML is older and more expressive, with attributes, namespaces and a long tradition of schema languages. YAML was revised in version 1.2 to be a strict superset of JSON, adding the comments and lighter punctuation people want in files they edit by hand. Binary encodings such as Protocol Buffers and CBOR trade readability for size and speed.

Why it matters

Picking JSON is really picking a contract between two systems: the shape of what crosses the wire, and sometimes the shape of what lands in storage.

On the wire it is the default because everything already speaks it. A browser parses it without a library, every mainstream language has a mature parser, and an engineer reading a log can see what a request carried. The costs lie elsewhere. Every record repeats its field names, numbers travel as decimal text that has to be parsed back, and the format itself says nothing about which fields are required or what type each one holds. A binary, schema-first encoding such as Protocol Buffers is smaller and faster to parse, and its contract is checked by a compiler, but a person needs a tool to read a single message. For a public or browser-facing API, JSON is the sensible choice. Between internal services where volume or latency dominates, the trade can go the other way.

A schema usually supplies the missing contract. JSON Schema declares which properties an object must have, what types they hold and whether extra properties are allowed, and OpenAPI builds on it to describe whole HTTP APIs. A schema is a separate document, and it stays accurate only while something forces it to match the code.

In storage the question is whether a structure belongs in a JSON column or in real columns. PostgreSQL offers two types. json keeps an exact copy of the input text and reparses it every time a function touches it, while jsonb stores a decomposed binary form that is slightly slower to write, much faster to query and can be indexed; its documentation recommends jsonb for most applications. A JSON column suits data whose shape varies from row to row or belongs to the client, such as a user’s saved preferences. A field the database has to join on, constrain or sum deserves a column of its own.

What it looks like in practice

On one platform the PostgreSQL functions assemble each complete response as JSON themselves, and the Go API forwards that JSON from the database to the browser untouched. The handler takes it as a json.RawMessage and passes it on without decoding or re-encoding it, and a small QueryJSON helper made that the default. The DTOs, mappers and response structs of a layered API fell away. When a function’s output changes, the new shape reaches the client with no edit to handler code.

Stored JSON brings a different problem, because it outlives the code that wrote it. One platform keeps notification settings and similar preferences as a user’s saved values laid over a set of defaults, originally merged one level deep. A key added to the defaults later was absent from every row saved before it existed, so client code read an undefined field and crashed, and it crashed for the longest-standing users. The fix was a deep merge that treats the defaults as a catalogue: every key in the catalogue comes out present, nested keys included, whatever the saved blob holds. A migration applied it to existing rows at once, and the frontend dropped its defensive checks.

JSON is also simply how data arrives. For historical electricity data delivered as JSONL, one JSON event per line, the task was to turn raw events into hourly consumption per zone, and three variants were built, in Python, JQ and SQL. Python was the fastest, at 34ms for one day of data and 180ms for thirty days.

Where it goes wrong

The grammar is small enough that parsers agree on what valid text looks like. Trouble comes from everything the grammar leaves open.

A large ID comes back one digit different. The grammar puts no ceiling on numbers, but most parsers, JavaScript’s among them, read them into 64-bit floating point, which holds integers exactly only up to 2⁵³ − 1. RFC 8259 names the range from −(2⁵³ − 1) to 2⁵³ − 1 as the one implementations agree on. Past it, identifiers change without a warning:

JSON.parse('{"id": 9007199254740993}').id;
// 9007199254740992

Send 64-bit identifiers as strings, and send money as a string or as an integer count of the smallest unit, never as a float.

Two services read the same object differently. RFC 8259 says the names in an object should be unique and leaves the rest open: given a repeated name, many parsers keep the last value, others raise an error, and some hand back every pair. When a gateway checks the first role and the backend acts on the second, the check protected nothing. Reject duplicate names where data enters the system. I-JSON, defined in RFC 7493, is a profile of JSON that forbids them, and a contract can cite it by name.

A hand-edited file stops parsing. Comments and trailing commas are outside the grammar, so the first person to annotate a config file, or to leave a comma after the last item, breaks it. Some tools accept a relaxed dialect, which makes the same file valid for one reader and broken for the next. Keep JSON for data machines write, and give hand-maintained files a format made for editing, such as YAML or TOML.

Text arrives garbled, or one parser rejects what another accepts. RFC 8259 requires UTF-8 between systems and forbids a byte order mark at the start. A producer that writes Latin-1, or an editor that prepends a mark, yields text whose fate depends on the parser. The grammar also allows an escape such as "\uDEAD", half of a surrogate pair with its partner missing, which no valid Unicode string can hold, and libraries treat it in different ways. Encode as UTF-8 at the source and reject invalid sequences at the boundary.

One small request ties up the parser. A few kilobytes of nested brackets can drive a recursive parser thousands of levels deep, and an oversized body is parsed in full before any schema sees it. RFC 8259 lets an implementation limit both text size and nesting depth, and an API that accepts JSON from strangers should use both: cap the request size before reading, cap the depth while parsing, and check the shape only after that.

Further reading

Glossary