{"canonical_url":"https://engineer.company/glossary/data-mesh/","content_markdown":"# Data mesh\n\n## What it is\n\nA data mesh is a way of organising analytical data in a large organisation so\nthat the teams who run each part of the business also own, publish and answer\nfor the data that part produces. Zhamak Dehghani, then a principal technology\nconsultant at Thoughtworks, set the idea out on Martin Fowler's site in a series\nof instalments in May 2019, and restated it in December 2020 as four\nprinciples: domain-oriented ownership of data and its architecture, data as a\nproduct, self-serve data infrastructure as a platform, and federated\ncomputational governance. She meant the four to be necessary and sufficient\ntogether, and expected the practices and the tools to change while the\nprinciples stay put.\n\nThe mesh is about analytical data, not the operational kind. Dehghani draws the\nline like this. Operational data sits in the databases behind the systems that\nrun the business, is transactional and holds the current state. Analytical data\nis the history of the business over time, aggregated and modelled, and it is\nwhat reports are built on and models are trained with. She leaves operational\ndata where it is, inside each service, and aims the mesh at the analytical side,\nwhich is where she sees the friction at scale.\n\n**Domain ownership.** The mesh is cut along lines the organisation already has.\nDehghani's running example is a digital media company with domains such as\npodcasts and artists. The team that runs podcasts, and offers the interfaces for\npublishing them, should also provide the history of released podcasts and how\nmany people listened over time. The boundary comes from domain-driven design,\nwhere each domain's bounded context is the place in which change stays local, so\nit is a good place to put ownership of the data as well.\n\n**Data as a product.** Spreading data across many teams makes it harder to find\nand to trust, so each team has to treat the data it shares as a product and the\npeople who use it as customers. The 2019 article lists what that takes: a data\nproduct should be discoverable, addressable, trustworthy, self-describing,\ninteroperable under global standards, and secured by global access control. A\ndomain data product owner answers for measures such as quality, how long\nconsumers wait for data and how satisfied they are. Responsibility for quality\nmoves upstream, as close to the source as it can get. In the architecture, a\ndata product is the smallest unit that can be deployed on its own, and it\ncarries three parts: the code that transforms and serves the data and enforces\nits policies, the data together with its metadata, and the infrastructure that\nruns it.\n\n**A self-serve platform.** Building, running and watching a data product takes\nspecialised infrastructure skills that cannot be copied into every domain. A\nplatform team therefore offers tools at a higher level, covering things like\nstorage, product schemas, declared pipelines and lineage, so that a generalist\ndeveloper in a domain can build a data product without that specialist\nknowledge.\n\n**Federated computational governance.** The products must still fit together, or\nnobody can combine them. A group of domain product owners and platform owners\ndecides which rules apply everywhere and which are left to each domain, and the\nplatform enforces the global ones automatically. Dehghani's example is listening\nfigures: how the podcast domain models them is its own business, but how a\nlistener is identified is a global matter, because the same person turns up as a\nuser in other domains too.\n\n## Why it matters\n\nDehghani's case begins with what goes wrong when a large company runs a central\ndata lake or warehouse. Data from every domain is pulled into one platform that\nowns all of it, run by a team of specialists in data tooling who sit apart from\nthe people who create the data and from the people who use it. She names three\nfailure modes. The platform is centralised and monolithic, so it struggles as\nsources and consumers multiply. It is split into pipeline stages such as\ningestion, cleansing, aggregation and serving, so a single new figure, like how\noften podcasts are played, needs a change in every stage and a release\ncoordinated across teams. And its ownership is siloed and hyper-specialised: the\nengineers who must make sense of the data know little of the domains it comes\nfrom, while the teams that produce it have no incentive to get it right.\n\nThe mesh changes who does the cleaning. In the old model, data arrives with its\nerrors and a central team cleanses it after ingestion. In a mesh, a domain's\ndataset becomes a data product only after the domain itself has checked it\nagainst the agreed quality measures and the global rules. The people who know\nwhat a field ought to mean are the ones who vouch for it.\n\nGovernance changes shape as well. The 2020 article sets the two models side by\nside in a table. A central team is itself responsible for quality, security and\ncompliance; a federated team defines how quality is modelled and which security\nand regulatory rules the platform should build in and monitor. Success used to\nbe counted in governed tables; in a mesh it is counted in how much the products\nare actually consumed. Manual review that tries to prevent errors gives way to\nautomated processes that detect them and recover.\n\nNone of this is offered as a universal answer. In the 2019 article Dehghani\ngrants that the central model can work for organisations with a simpler domain\nand fewer kinds of consumer, and says it fails for enterprises with rich\ndomains, many sources and a diverse set of consumers. Whether a mesh is worth\nits overhead depends on which of the two an organisation is.\n\n## What it looks like in practice\n\nA real mesh spans many teams and many stores. The sketch below keeps to one\nPostgreSQL database, so that each statement can be checked against the manual,\nand shows the two smallest moves: a domain publishing a product, and the\nplatform enforcing one global rule.\n\nThe orders team owns the operational tables in the `orders` schema. It publishes\nits analytical data in a schema of its own, as a versioned view with a\ndescription and an explicit grant:\n\n```sql\n-- the orders team owns this schema and everything in it\nCREATE SCHEMA order_data;\n\nCREATE VIEW order_data.orders_by_day_v1 AS\nSELECT order_date, currency, count(*) AS orders, sum(total) AS revenue\nFROM orders.orders\nWHERE status = 'paid'\nGROUP BY order_date, currency;\n\nCOMMENT ON VIEW order_data.orders_by_day_v1 IS\n  'Paid orders per day and currency. Owner: orders team.';\n\nGRANT USAGE ON SCHEMA order_data TO analysts;\nGRANT SELECT ON order_data.orders_by_day_v1 TO analysts;\n```\n\nThe PostgreSQL manual explains why this is enough on the consumer's side:\nwhoever uses a view needs the USAGE privilege only on the schema that holds the\nview, not on the schemas the view reads from. The analysts can query the product\nwhile the operational tables behind it stay closed to them. Evans, for his part,\nlists database schemas among the physical forms a bounded context's boundary can\ntake.\n\nThe platform's side is a rule that holds for every domain, written as a query\nand run on every release: no product schema may contain a view without a\ndescription. `obj_description` returns the comment stored on an object, and a\n`relkind` of `v` marks a view in `pg_class`:\n\n```sql\n-- global rule: every published view carries a description\nSELECT n.nspname, c.relname\nFROM pg_class AS c\nJOIN pg_namespace AS n ON n.oid = c.relnamespace\nWHERE c.relkind = 'v'\n  AND n.nspname ~ '_data$'\n  AND obj_description(c.oid, 'pg_class') IS NULL;\n-- any row returned stops the release\n```\n\nThat split is the federated part in miniature. What counts as a paid order is\nthe orders team's decision. That every product describes itself, and that\nproduct schemas end in `_data`, are global conventions the platform checks\nwithout anybody having to remember them.\n\nThe service page [Data Governance \u0026 Quality](/services/data-governance/)\npromises that it will be plain who owns each figure and what its source is,\nwhich is exactly what a data product owner must be able to say. Among its\ndelivered work it lists a function-first PostgreSQL data layer, designed so that\nreads and writes pass through functions the database can grant access to, never\nthrough the tables themselves. It is the instinct behind the view above:\nconsumers get an interface with a grant on it, not the tables underneath.\n\nThe portfolio case\n[One infrastructure plan for 14 departments](/portfolio/one-infrastructure-plan-for-14-departments/)\ndoes not call itself a data mesh, but it runs into the tension a mesh's\ngovernance group lives with. At a research institute, 14 research groups worked\nwith different data sources, formats, scales and tools, and their technical\nskills and IT resources differed widely: a few had built their own solutions,\nwhile many struggled. The plan set out to let researchers concentrate on\nresearch, and it balanced flexibility against standardisation, with a modular\narchitecture, ways in for very different data sources, interfaces suited to\ndifferent levels of skill, scalable storage and processing, and strategies for\nonboarding, support and governance. Deciding what each group keeps for itself\nand what everyone shares is the balance Dehghani gives to federated governance,\nand lowering the skill a group needs is what she asks of a self-serve platform.\n\n## Where it goes wrong\n\n**A central team with a new name.** Calling the data platform team a mesh\nchanges nothing if the same team still takes in every domain's raw data, cleans\nit and answers for its quality. Dehghani's governance table makes a usable\nchecklist: who is responsible for quality, who decides the security rules, and\nwhether success is still counted in governed tables. Move ownership of each data\nproduct into the domain that produces the data, and keep the central team for\nthe platform and the rules.\n\n**Products nobody stands behind.** A domain that exports tables without an\nowner, a description or a promise about freshness has only moved the old problem\ninto its own corner. Dehghani asks each data product to state its quality as\nservice level objectives, such as timeliness and error rates, and notes that one\ndomain may offer two products of the same events: one close to real time that\ncan miss or repeat events, and one that arrives later and is more accurate. Name\nan owner for every product, publish its objectives, and test quality where the\ndata is produced.\n\n**No shared identifiers.** If each domain identifies the same things its own\nway, no two products can be joined. In the 2019 example an artist looks\ndifferent in the domain that records plays and in the domain that pays artists.\nAgree globally on how such shared entities, which Dehghani calls polysemes, are\nidentified, and leave the rest of each model to its domain. Field formats,\naddress conventions and common metadata fields belong on the same global list.\n\n**Every domain rebuilds the stack.** Without a self-serve platform, each domain\nhas to provision its own storage, pipelines and monitoring, and the specialised\nskills that were scarce in one central team are now needed in every team. Build\nthe platform before asking domains to publish, and judge it by how little\nspecialist knowledge a domain needs to ship a product.\n\n**Changing a product under its consumers.** A data product is an interface, and\nchanging it in place breaks the reports built on it. In PostgreSQL, `CREATE OR\nREPLACE VIEW` refuses part of this mistake: the new query must produce the same\ncolumns, with the same names, order and data types, and may only add columns at\nthe end. It does not catch a change of meaning, because the calculation behind a\ncolumn may be completely different. Put any change that renames, drops or\nredefines a column into a new view, `orders_by_day_v2`, beside the old one, and\nretire the old one once its consumers have moved.\n\n**A mesh where none is needed.** Four principles, a platform team, a federated\ngovernance group and a product owner in every domain are a lot of machinery.\nNear its end, Dehghani's 2019 article admits that \"enterprise scale adoption\nstill has a long way to go\". If a handful of sources and one team cover the\nneeds, keep a well-run central store with a named owner for each table, and\nborrow the parts that pay at any size: owners, described datasets and rules that\nthe machine checks.\n\n## Further reading\n\n- [Data Mesh Principles and Logical Architecture](https://martinfowler.com/articles/data-mesh-principles.html)\n  — Zhamak Dehghani, 2020: the four principles, the parts of a data product and\n  the table that contrasts central with federated governance.\n- [How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh](https://martinfowler.com/articles/data-monolith-to-mesh.html)\n  — the original article from May 2019, with the three failure modes of a\n  central data platform and the qualities a data product needs.\n- [Domain-Driven Design Reference](https://www.domainlanguage.com/wp-content/uploads/2016/05/DDD_Reference_2015-03.pdf)\n  — Eric Evans's summary of the patterns, including the bounded context along\n  whose edges a mesh divides ownership.\n- [PostgreSQL: CREATE VIEW](https://www.postgresql.org/docs/current/sql-createview.html)\n  — the rules for replacing a view and the privileges its users need, as used in\n  the example.\n- [PostgreSQL: COMMENT](https://www.postgresql.org/docs/current/sql-comment.html)\n  — how a description is stored on a database object, which `obj_description`\n  then reads back.\n\n\u003chttps://engineer.company/glossary/data-mesh/\u003e\n","date_modified":"2026-10-29T11:17:41Z","description":"How a data mesh gives analytical data to the teams that produce it, serves it as products on a shared platform, and where the idea goes wrong.","formats":{"gemtext":"https://engineer.company/glossary/data-mesh/index.gmi","gopher":"https://engineer.company/glossary/data-mesh/gophermap","html":"https://engineer.company/glossary/data-mesh/","json":"https://engineer.company/glossary/data-mesh/index.json","markdown":"https://engineer.company/glossary/data-mesh/index.md","pdf":"https://engineer.company/glossary/data-mesh/index.pdf"},"kind":"page","language":"en","title":"What is a data mesh?","url":"https://engineer.company/glossary/data-mesh/","version":1}