{"canonical_url":"https://engineer.company/glossary/data-lake/","content_markdown":"# Data lake\n\n## What it is\n\nA data lake is a store that keeps raw data as it arrives from its source, at the\nfinest level of detail the source provides, so that questions nobody has asked\nso far can still be answered from it later. In practice the data sits in files,\noften in a column-oriented format such as Apache Parquet, in a distributed file\nsystem or an object store, and query engines read those files directly.\n\nThe term comes from James Dixon of Pentaho. In a blog post dated 14 October\n2010, written the week Pentaho announced its first Hadoop release, he summed up\nwhat he had heard from companies that used Hadoop. Eight or nine in ten of them\ndealt with data that was structured or semi-structured rather than unstructured,\nand it usually came from a single application or system. A few questions about\nit were known, many more would come up later, several groups of users wanted to\nask them, and the volume would not fit into a relational database, either\ntechnically or at a sensible cost. The usual answer had been to pick the most\ninteresting attributes and aggregate them into a data mart. That answers only\nthe questions chosen in advance, and the aggregation hides the lowest level of\ndetail for good.\n\nDixon offered a picture instead. A data mart is like bottled water: cleaned,\npackaged and given a shape that makes it easy to consume. A data lake, in his\nwords, is \"a large body of water in a more natural state\". One source keeps it\nfilled, and anyone with a question can come and look, wade in or draw a sample.\n\nFour years later Dixon revisited the term, because it was being used with many\ndifferent definitions. He restated his own: a lake holds data from one source,\nnot pre-aggregated, and an organisation can have several lakes. Data from many\nsystems joined together was, in his image, a water garden rather than a lake. He\nwas equally clear that a lake did not replace a database, a data mart or a data\nwarehouse in 2010. Several of the definitions he was answering described one\nshared store for raw data from many systems. That is the looser sense of the\nword, and it is worth knowing which one a colleague means.\n\n## Why it matters\n\nEvery summary is a decision about which questions matter. A daily sales total\nper shop tells you how much each shop sold. It cannot tell you which hour was\nbusiest, which products people looked at and did not buy, or whether one\ncustomer came back three times before ordering. Once only the total is kept,\nthose questions are gone for good, because the rows that could answer them were\nthrown away when the total was computed. Dixon's argument was that questions\narrive after the data does, so the raw records should be kept at their most\ngranular level and aggregated once the question is known.\n\nThat does not make the warehouse obsolete. Dixon described the institutional\ndata marts and warehouses being populated with aggregations taken from the lake,\nwhile ad-hoc questions go to the lake itself. The warehouse remains the place\nfor the figures the organisation has agreed on. The lake is where you go when a\nnew question needs detail the warehouse never kept, and where you rebuild a\nwarehouse table when its definition changes.\n\nThe file format matters as much as the idea. Parquet stores each column of a row\ngroup as its own chunk, and the metadata at the end of the file records where\nevery chunk starts. A reader looks at that metadata first and then fetches only\nthe column chunks it needs, so a query that touches three columns out of forty\ndoes not read the other thirty-seven. Because Parquet is an open format and each\nfile carries its own metadata, any engine that reads Parquet can query the same\nfolder, DuckDB on a laptop included, without an export step.\n\n## What it looks like in practice\n\nA web shop records every product view, add-to-cart and purchase. Instead of\nsumming them into daily totals, it keeps every event and writes the events out\nas Parquet files, one folder per day. In DuckDB 2.0, both steps are plain SQL:\n\n```sql\nCREATE TABLE shop_events AS\nSELECT * FROM (VALUES\n  (TIMESTAMP '2026-10-26 09:14:03', 'view',     'boots-41', NULL::INTEGER),\n  (TIMESTAMP '2026-10-26 09:15:40', 'cart',     'boots-41', NULL),\n  (TIMESTAMP '2026-10-26 09:17:12', 'purchase', 'boots-41', 1200),\n  (TIMESTAMP '2026-10-27 11:02:55', 'view',     'jacket-m', NULL),\n  (TIMESTAMP '2026-10-27 11:03:30', 'view',     'boots-41', NULL),\n  (TIMESTAMP '2026-10-27 11:05:01', 'cart',     'jacket-m', NULL)\n) t(occurred_at, event, product, amount);\n\nCOPY (SELECT *, CAST(occurred_at AS DATE) AS day FROM shop_events)\nTO 'events' (FORMAT parquet, PARTITION_BY (day));\n```\n\n`PARTITION_BY` accepts column names only, not expressions, so the day is\ncomputed in the `SELECT`. DuckDB writes one folder per value, named in the\nkey=value form its documentation calls Hive partitioning:\n\n```text\nevents/day=2026-10-26/data_0.parquet\nevents/day=2026-10-27/data_0.parquet\n```\n\nA question about one day filters on that column:\n\n```sql\nSELECT event, count(*) AS n\nFROM   read_parquet('events/*/*.parquet', hive_partitioning = true)\nWHERE  day = DATE '2026-10-27'\nGROUP  BY event\nORDER  BY n DESC;\n-- view 2, cart 1\n```\n\nDuckDB pushes a filter on a partition column down to the folder names and skips\nthe files that cannot match. `EXPLAIN ANALYZE` on this query reports one of the\ntwo files scanned. With a year of daily folders, the same question would open\none folder out of 365. The `day` column comes back as a `DATE` without being\ntold, since DuckDB detects dates, timestamps and big integers in folder names by\ndefault. It also switches Hive partitioning on by itself when the folders follow\nthe key=value pattern, so the flag in these queries only makes the intent\nvisible.\n\nNow a question nobody planned for: which products were looked at most?\n\n```sql\nSELECT product, count(*) AS views\nFROM   read_parquet('events/*/*.parquet', hive_partitioning = true)\nWHERE  event = 'view'\nGROUP  BY product\nORDER  BY views DESC, product;\n-- boots-41 2, jacket-m 1\n```\n\nA table of daily purchase totals could never answer that. The raw events answer\nit with one more query against the same folder, because every event was kept on\nthe way in.\n\nThe company's [data warehouse design service](/services/data-warehousing/)\ndescribes its aim as one place where all the numbers agree, so that two\ndepartments stop bringing two answers to the same meeting. A lake like this one\nsits upstream of such a warehouse: the warehouse holds the agreed figures, and\nthe raw events stay on hand for the next question.\n\n## Where it goes wrong\n\n**Files nobody can explain.** A folder of Parquet files with no record of where\nthey came from, who owns them or what each column means is a lake only its\nbuilder can use. When critics argued in 2014 that a lake has no metadata, Dixon\nreplied that nothing in the idea says so, and later added that anyone joining\ndata from different sources or tracing lineage will need metadata. Keep one\ncatalogue entry per dataset with its source, owner, columns, load time and\nretention, and make writing that entry part of the load job.\n\n**Thousands of tiny files.** Every Parquet file carries its own metadata block,\nand a reader has to read it before any data. A stream that writes one small file\na minute leaves more than 40,000 files for a 30-day month, each opened and\nparsed on every full scan. Batch writes into fewer, larger files, and compact\nold partitions on a schedule. The Apache Iceberg table format records such a\ncompaction as a `replace` snapshot: several files swapped for one that holds the\nsame rows.\n\n**Readers that see half a load.** A job that writes ten files and stops after\nsix leaves six files in the folder, and anything that lists the folder will read\nthem. A plain directory has no commit. A table format fixes this. Iceberg tracks\nindividual data files in metadata rather than whole directories and commits a\nchange by atomically swapping one metadata file for the next. Readers always use\na committed snapshot, writes are never partially visible, and readers take no\nlocks.\n\n**Renamed columns that split the history.** Old files keep the old column name,\nand every query has to know both. Iceberg selects columns in data files by a\nfield id instead of by name, so a column can be renamed, added, dropped or\nmoved, even inside nested structures, without rewriting the files already\nwritten.\n\n**One lake for every system.** Pouring every database in the organisation into\none place and joining across all of it is what Dixon called a water garden, and\nhe advocated using several kinds of storage across an enterprise, each chosen\nfor the data and queries it handles. His advice was to use the best tool for\neach job: a transactional database for transactional purposes and an analytic\ndatabase for analytic ones. Start a lake from one source with real questions\nwaiting, and build the integrated, agreed figures in a warehouse populated from\nit.\n\n**Personal data kept forever.** A store designed to keep everything collides\nwith deletion requests and retention limits. Iceberg version 2 can delete\nindividual rows by writing delete files, without rewriting the data files, which\nmeans the original bytes stay in storage. They go only when the data files are\nrewritten and the older snapshots that still list them expire, after which the\nfiles may be physically deleted. Keep direct identifiers out of raw events where\nyou can, partition so that retention drops whole folders, and schedule\ncompaction and snapshot expiry after deletions.\n\n## Further reading\n\n- [Pentaho, Hadoop, and Data Lakes](https://jamesdixon.wordpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/)\n  — James Dixon's October 2010 post that introduced the term and the comparison\n  with bottled water.\n- [Data Lakes Revisited](https://jamesdixon.wordpress.com/2014/09/25/data-lakes-revisited/)\n  — Dixon's 2014 follow-up: one source per lake, the water garden, and his\n  answers to the critics.\n- [File Format](https://parquet.apache.org/docs/file-format/) — the Parquet\n  layout of row groups, column chunks and file metadata written after the data.\n- [Iceberg Table Spec](https://iceberg.apache.org/spec/) — the Apache Iceberg\n  specification: snapshots, atomic commits, schema evolution by field id, delete\n  files and snapshot expiry.\n- [Hive Partitioning](https://duckdb.org/docs/2.0/data/partitioning/hive_partitioning.html)\n  — DuckDB's documentation of key=value folders, filter pushdown and partition\n  type detection.\n\n\u003chttps://engineer.company/glossary/data-lake/\u003e\n","date_modified":"2026-10-30T02:51:31Z","description":"What a data lake is: James Dixon's 2010 idea, raw data kept at full detail, a partitioned Parquet example in DuckDB, and where lakes go wrong.","formats":{"gemtext":"https://engineer.company/glossary/data-lake/index.gmi","gopher":"https://engineer.company/glossary/data-lake/gophermap","html":"https://engineer.company/glossary/data-lake/","json":"https://engineer.company/glossary/data-lake/index.json","markdown":"https://engineer.company/glossary/data-lake/index.md","pdf":"https://engineer.company/glossary/data-lake/index.pdf"},"kind":"page","language":"en","title":"What is a data lake?","url":"https://engineer.company/glossary/data-lake/","version":1}