Data mesh

What it is

A data mesh is a way of organising analytical data in a large
organisation so that the teams who run each part of the business
also own, publish and answer for the data that part produces. Zhamak
Dehghani, then a principal technology consultant at Thoughtworks,
set the idea out on Martin Fowler's site in a series of instalments
in May 2019, and restated it in December 2020 as four principles:
domain-oriented ownership of data and its architecture, data as a
product, self-serve data infrastructure as a platform, and federated
computational governance. She meant the four to be necessary and
sufficient together, and expected the practices and the tools to
change while the principles stay put.

The mesh is about analytical data, not the operational kind.
Dehghani draws the line like this. Operational data sits in the
databases behind the systems that run the business, is transactional
and holds the current state. Analytical data is the history of the
business over time, aggregated and modelled, and it is what reports
are built on and models are trained with. She leaves operational
data where it is, inside each service, and aims the mesh at the
analytical side, which is where she sees the friction at scale.

Domain ownership. The mesh is cut along lines the organisation
already has. Dehghani's running example is a digital media company
with domains such as podcasts and artists. The team that runs
podcasts, and offers the interfaces for publishing them, should also
provide the history of released podcasts and how many people
listened over time. The boundary comes from domain-driven design,
where each domain's bounded context is the place in which change
stays local, so it is a good place to put ownership of the data as
well.

Data as a product. Spreading data across many teams makes it harder
to find and to trust, so each team has to treat the data it shares
as a product and the people who use it as customers. The 2019
article lists what that takes: a data product should be
discoverable, addressable, trustworthy, self-describing,
interoperable under global standards, and secured by global access
control. A domain data product owner answers for measures such as
quality, how long consumers wait for data and how satisfied they
are. Responsibility for quality moves upstream, as close to the
source as it can get. In the architecture, a data product is the
smallest unit that can be deployed on its own, and it carries three
parts: the code that transforms and serves the data and enforces its
policies, the data together with its metadata, and the
infrastructure that runs it.

A self-serve platform. Building, running and watching a data product
takes specialised infrastructure skills that cannot be copied into
every domain. A platform team therefore offers tools at a higher
level, covering things like storage, product schemas, declared
pipelines and lineage, so that a generalist developer in a domain
can build a data product without that specialist knowledge.

Federated computational governance. The products must still fit
together, or nobody can combine them. A group of domain product
owners and platform owners decides which rules apply everywhere and
which are left to each domain, and the platform enforces the global
ones automatically. Dehghani's example is listening figures: how the
podcast domain models them is its own business, but how a listener
is identified is a global matter, because the same person turns up
as a user in other domains too.

Why it matters

Dehghani's case begins with what goes wrong when a large company
runs a central data lake or warehouse. Data from every domain is
pulled into one platform that owns all of it, run by a team of
specialists in data tooling who sit apart from the people who create
the data and from the people who use it. She names three failure
modes. The platform is centralised and monolithic, so it struggles
as sources and consumers multiply. It is split into pipeline stages
such as ingestion, cleansing, aggregation and serving, so a single
new figure, like how often podcasts are played, needs a change in
every stage and a release coordinated across teams. And its
ownership is siloed and hyper-specialised: the engineers who must
make sense of the data know little of the domains it comes from,
while the teams that produce it have no incentive to get it right.

The mesh changes who does the cleaning. In the old model, data
arrives with its errors and a central team cleanses it after
ingestion. In a mesh, a domain's dataset becomes a data product only
after the domain itself has checked it against the agreed quality
measures and the global rules. The people who know what a field
ought to mean are the ones who vouch for it.

Governance changes shape as well. The 2020 article sets the two
models side by side in a table. A central team is itself responsible
for quality, security and compliance; a federated team defines how
quality is modelled and which security and regulatory rules the
platform should build in and monitor. Success used to be counted in
governed tables; in a mesh it is counted in how much the products
are actually consumed. Manual review that tries to prevent errors
gives way to automated processes that detect them and recover.

None of this is offered as a universal answer. In the 2019 article
Dehghani grants that the central model can work for organisations
with a simpler domain and fewer kinds of consumer, and says it fails
for enterprises with rich domains, many sources and a diverse set of
consumers. Whether a mesh is worth its overhead depends on which of
the two an organisation is.

What it looks like in practice

A real mesh spans many teams and many stores. The sketch below keeps
to one PostgreSQL database, so that each statement can be checked
against the manual, and shows the two smallest moves: a domain
publishing a product, and the platform enforcing one global rule.

The orders team owns the operational tables in the `orders` schema.
It publishes its analytical data in a schema of its own, as a
versioned view with a description and an explicit grant:

```sql
i-- the orders team owns this schema and everything in it		engineer.company	70
CREATE SCHEMA order_data;

CREATE VIEW order_data.orders_by_day_v1 AS
SELECT order_date, currency, count(*) AS orders, sum(total) AS revenue
FROM orders.orders
WHERE status = 'paid'
GROUP BY order_date, currency;

COMMENT ON VIEW order_data.orders_by_day_v1 IS
  'Paid orders per day and currency. Owner: orders team.';

GRANT USAGE ON SCHEMA order_data TO analysts;
GRANT SELECT ON order_data.orders_by_day_v1 TO analysts;
```

The PostgreSQL manual explains why this is enough on the consumer's
side: whoever uses a view needs the USAGE privilege only on the
schema that holds the view, not on the schemas the view reads from.
The analysts can query the product while the operational tables
behind it stay closed to them. Evans, for his part, lists database
schemas among the physical forms a bounded context's boundary can
take.

The platform's side is a rule that holds for every domain, written
as a query and run on every release: no product schema may contain a
view without a description. `obj_description` returns the comment
stored on an object, and a `relkind` of `v` marks a view in
`pg_class`:

```sql
i-- global rule: every published view carries a description		engineer.company	70
SELECT n.nspname, c.relname
FROM pg_class AS c
JOIN pg_namespace AS n ON n.oid = c.relnamespace
WHERE c.relkind = 'v'
  AND n.nspname ~ '_data$'
  AND obj_description(c.oid, 'pg_class') IS NULL;
i-- any row returned stops the release		engineer.company	70
```

That split is the federated part in miniature. What counts as a paid
order is the orders team's decision. That every product describes
itself, and that product schemas end in `_data`, are global
conventions the platform checks without anybody having to remember
them.

The service page Data Governance & Quality promises that it will be
plain who owns each figure and what its source is, which is exactly
what a data product owner must be able to say. Among its delivered
work it lists a function-first PostgreSQL data layer, designed so
that reads and writes pass through functions the database can grant
access to, never through the tables themselves. It is the instinct
behind the view above: consumers get an interface with a grant on
it, not the tables underneath.

The portfolio case One infrastructure plan for 14 departments does
not call itself a data mesh, but it runs into the tension a mesh's
governance group lives with. At a research institute, 14 research
groups worked with different data sources, formats, scales and
tools, and their technical skills and IT resources differed widely:
a few had built their own solutions, while many struggled. The plan
set out to let researchers concentrate on research, and it balanced
flexibility against standardisation, with a modular architecture,
ways in for very different data sources, interfaces suited to
different levels of skill, scalable storage and processing, and
strategies for onboarding, support and governance. Deciding what
each group keeps for itself and what everyone shares is the balance
Dehghani gives to federated governance, and lowering the skill a
group needs is what she asks of a self-serve platform.

Where it goes wrong

A central team with a new name. Calling the data platform team a
mesh changes nothing if the same team still takes in every domain's
raw data, cleans it and answers for its quality. Dehghani's
governance table makes a usable checklist: who is responsible for
quality, who decides the security rules, and whether success is
still counted in governed tables. Move ownership of each data
product into the domain that produces the data, and keep the central
team for the platform and the rules.

Products nobody stands behind. A domain that exports tables without
an owner, a description or a promise about freshness has only moved
the old problem into its own corner. Dehghani asks each data product
to state its quality as service level objectives, such as timeliness
and error rates, and notes that one domain may offer two products of
the same events: one close to real time that can miss or repeat
events, and one that arrives later and is more accurate. Name an
owner for every product, publish its objectives, and test quality
where the data is produced.

No shared identifiers. If each domain identifies the same things its
own way, no two products can be joined. In the 2019 example an
artist looks different in the domain that records plays and in the
domain that pays artists. Agree globally on how such shared
entities, which Dehghani calls polysemes, are identified, and leave
the rest of each model to its domain. Field formats, address
conventions and common metadata fields belong on the same global
list.

Every domain rebuilds the stack. Without a self-serve platform, each
domain has to provision its own storage, pipelines and monitoring,
and the specialised skills that were scarce in one central team are
now needed in every team. Build the platform before asking domains
to publish, and judge it by how little specialist knowledge a domain
needs to ship a product.

Changing a product under its consumers. A data product is an
interface, and changing it in place breaks the reports built on it.
In PostgreSQL, `CREATE OR REPLACE VIEW` refuses part of this
mistake: the new query must produce the same columns, with the same
names, order and data types, and may only add columns at the end. It
does not catch a change of meaning, because the calculation behind a
column may be completely different. Put any change that renames,
drops or redefines a column into a new view, `orders_by_day_v2`,
beside the old one, and retire the old one once its consumers have
moved.

A mesh where none is needed. Four principles, a platform team, a
federated governance group and a product owner in every domain are a
lot of machinery. Near its end, Dehghani's 2019 article admits that
"enterprise scale adoption still has a long way to go". If a handful
of sources and one team cover the needs, keep a well-run central
store with a named owner for each table, and borrow the parts that
pay at any size: owners, described datasets and rules that the
machine checks.

Further reading

  * Data Mesh Principles and Logical Architecture — Zhamak Dehghani,
2020: the four principles, the parts of a data product and the table
that contrasts central with federated governance.

  * How to Move Beyond a Monolithic Data Lake to a Distributed Data
Mesh — the original article from May 2019, with the three failure
modes of a central data platform and the qualities a data product
needs.

  * Domain-Driven Design Reference — Eric Evans's summary of the
patterns, including the bounded context along whose edges a mesh
divides ownership.

  * PostgreSQL: CREATE VIEW — the rules for replacing a view and the
privileges its users need, as used in the example.

  * PostgreSQL: COMMENT — how a description is stored on a database
object, which `obj_description` then reads back.

hThis page on the web	URL:https://engineer.company/glossary/data-mesh/	engineer.company	70
