{"canonical_url":"https://engineer.company/glossary/data-warehouse/","content_markdown":"# Data warehouse\n\n## What it is\n\nA data warehouse is a database built to answer questions about a business rather\nthan to run it. The systems that take orders, book shipments, issue invoices and\nlog support calls each keep their own data, shaped for their own work. A\nwarehouse receives copies from all of them, keeps those copies for years, and\narranges them so that a question spanning several systems becomes one query\ninstead of a week of exports. Building and running one is called data\nwarehousing.\n\nThe difference from the databases it copies from is in the workload. An order\nsystem reads and writes a few rows at a time, thousands of times a minute, and\nkeeps only the present: when a customer moves, the old address is overwritten. A\nwarehouse is written in batches, read by queries that scan millions of rows at\nonce, and keeps the past on purpose, so last year's figures can still be broken\ndown the way the business looked last year.\n\nRalph Kimball brought dimensional modelling to the warehouse industry in 1996,\nand the Kimball Group divides such a system into two rooms. The back room holds\nthe ETL system, which they break into 34 subsystems grouped under four jobs:\nextracting data from the sources, cleaning and conforming it, delivering it to\nthe presentation area, and managing the whole process. The front room holds the\npresentation area, which is what people actually query. It is organised by\nbusiness process, kept at the most detailed level each process records, with\naggregated summaries added for speed, and structured as fact and dimension\ntables. Around it sit the BI applications: ad hoc queries, standard reports,\ndashboards, scorecards and analytic applications. The Kimball Group calls the\nfront room \"the public face of the DW/BI system\".\n\nThat is not the only arrangement. The Kimball Group's own architecture page\ndescribes the Corporate Information Factory, a hub-and-spoke design that keeps\natomic data in a normalised enterprise warehouse in third normal form and runs\na second ETL step to load dimensional tables for reporting, and a hybrid in\nwhich an existing normalised warehouse supplies a dimensional presentation area.\nWhat every arrangement shares is the separation: the systems that run the\nbusiness are not the place where it is analysed.\n\nA warehouse is a role, not a product. It can be a PostgreSQL or Microsoft SQL\nServer database on a server the team runs, or a managed service such as Google\nBigQuery, which Google describes as a fully managed data platform combining a\ncloud data warehouse with analytic tools. BigQuery keeps storage and compute\napart so each scales on its own, stores data by column rather than by row, and\ntakes queries in ANSI-standard SQL.\n\n## Why it matters\n\nA business that asks the same question of two systems gets two answers. Sales\ncounts a customer when an order is placed, finance when an invoice is paid, and\nsupport whenever a ticket arrives. None of them is wrong for its own purpose,\nand none of them adds up with the others. The warehouse is where one definition\nis agreed and written down in the load, so that every report built on it counts\nthe same way.\n\nThe Kimball Group's tool for this is the conformed dimension: one customer, one\nproduct, one calendar, maintained centrally in the ETL system and shared by\nevery fact table that needs it. A sales fact table and a support fact table that\nboth point at the same customer dimension can be compared line by line. Their\nbus architecture lays the business processes out against the dimensions they\nshare in a matrix they call the architectural blueprint, and it lets the\nwarehouse be built one business process at a time, which they say shortens the\ntime to market. A team can deliver orders this quarter and returns next quarter\nwithout redesigning what it already shipped, as long as both use the same\ndimensions.\n\nHistory is the second reason. Operational systems keep the current state; a\nwarehouse can keep every state. The Kimball technique for a changing attribute,\ncalled Type 2, adds a new dimension row with the new values and a new surrogate\nkey, and every fact loaded from that moment on points to the new row. Sales\nfrom before a customer moved stay with the old region; sales after go to the\nnew one. At least three extra columns, an effective date, an expiry date and a\ncurrent-row flag, make it possible to ask what was true at any date.\n\nThe third reason is plain load. A report that scans three years of orders does\nnot belong on the database that is taking today's orders, where it competes for\nthe same memory, disk and locks as the checkout. Moving analysis to its own\ndatabase lets each side be tuned for its own work.\n\nThe price is that a warehouse is a copy. It is as fresh as its last load and as\ncorrect as the rules in that load, and someone has to own both: the schedule\nthat runs it, and the definitions of a sale, a customer and a month that the\nbusiness agreed to.\n\n## What it looks like in practice\n\nThe heart of a dimensional warehouse is a fact table whose grain is declared\nfirst, here one row per order line, surrounded by the dimensions that describe\nit. The customer dimension carries the Type 2 columns, so a customer who moves\ngets a second row rather than an overwritten one.\n\n```sql\nCREATE TABLE dim_customer (\n    customer_key bigint PRIMARY KEY,   -- surrogate key, owned by the warehouse\n    customer_id  text NOT NULL,        -- the key in the source system\n    region       text NOT NULL,\n    valid_from   date NOT NULL,\n    valid_to     date,                 -- empty while the row is current\n    is_current   boolean NOT NULL\n);\n\nCREATE TABLE fact_order_line (\n    date_key     integer NOT NULL REFERENCES dim_date,\n    customer_key bigint  NOT NULL REFERENCES dim_customer,\n    product_key  bigint  NOT NULL REFERENCES dim_product,\n    quantity     integer NOT NULL,\n    net_amount   numeric(12, 2) NOT NULL\n);\n\nSELECT d.year, c.region, sum(f.net_amount) AS revenue\nFROM fact_order_line AS f\nJOIN dim_date AS d USING (date_key)\nJOIN dim_customer AS c USING (customer_key)\nGROUP BY d.year, c.region\nORDER BY d.year, c.region;\n```\n\nBecause each order line was loaded with the customer row that was current at\nthe time, the query reports revenue by the region the customer lived in when\nthey bought. Joining on `customer_id` and filtering on `is_current` instead\nrestates all of history by today's regions. Both are legitimate questions, and\nthe model answers either without a second copy of the data.\n\nThe promise of the [data warehouse design service](/services/data-warehousing/) is exactly that first reason: sales, operations and finance figures brought together in one place, so two departments stop bringing two answers to the same meeting. At larger scale, the founder [architected and ran 100 data warehouse databases](/portfolio/ran-100-postgresql-ms-sql-bigquery-warehouses/) on PostgreSQL, Microsoft SQL Server and Google BigQuery between 2022 and 2024, holding mostly GIS and time-series data such as spatial coordinates and sensor readings. The work combined normalised and denormalised table layouts with spatial indexing (PostGIS on PostgreSQL) and time-series partitioning on the first two engines, and time-partitioned, clustered tables with compression and columnar storage on BigQuery. Query response times for GIS and time-series data fell by 40 to 60%, and the platform let the organisation using those databases launch new data products such as real-time analytics dashboards.\n\n## Where it goes wrong\n\n**One fact table holds two grains, and every total counts some money twice.** An\norder has lines, and it also has a delivery charge that belongs to the order as\na whole. Put the charge on every line and a sum over lines counts it once per\nline. The Kimball Group's four-step design process puts declaring the grain\nsecond, straight after choosing the business process and before any dimension\nor fact is named, and their rule is that different grains must not share one\nfact table.\nKeep a separate fact table at order grain for charges that belong to the whole\norder, or divide the charge across the lines by a rule the business agrees to.\n\n**A customer moves, and last year's report changes.** When the region on the\ncustomer row is simply overwritten, every past sale follows the customer to the\nnew region, and a report that was signed off in January tells a different story\nin June. Treat attributes whose history matters with Type 2: close the current\nrow and add a new one with a new surrogate key, in one database transaction so\nno load ever sees both rows current.\n\n```sql\nBEGIN;\nUPDATE dim_customer\nSET valid_to = current_date, is_current = false\nWHERE customer_id = 'C-1042' AND is_current;\nINSERT INTO dim_customer\n    (customer_key, customer_id, region, valid_from, valid_to, is_current)\nVALUES (90731, 'C-1042', 'North', current_date, NULL, true);\nCOMMIT;\n```\n\n**Two teams build two customer dimensions, and the warehouse repeats the\nargument it was meant to end.** Each new subject area is quicker to deliver with\nits own copy of the customer or product list, and each copy drifts: one drops\nclosed accounts, the other merges duplicates differently. Build the shared\ndimensions once in the ETL system and make every new fact table use them. The\nbus matrix is the checklist: before a business process is added, it shows which\nexisting dimensions the new fact table has to point at.\n\n**Reports are pointed at the staging tables.** Raw copies of the sources land in\nthe back room first, and once they are there it is tempting to query them\ndirectly because they are already loaded. Those tables carry each source's own\ncodes, duplicates and gaps, and they change whenever a source does, so every\nreport built on them inherits the problem the warehouse was meant to solve.\nKeep the back room for the ETL system alone and give reports, dashboards and\nanalysts the presentation area only, where the conformed dimensions and the\ndeclared grain are what make the numbers agree.\n\n## Further reading\n\n- [Kimball Technical DW/BI System Architecture](https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/technical-dw-bi-system-architecture/)\n  — the back room and the front room, the ETL subsystems, metadata, and the\n  hub-and-spoke and hybrid alternatives.\n- [Enterprise Data Warehouse Bus Architecture](https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/kimball-data-warehouse-bus-architecture/)\n  — conformed dimensions, the bus matrix, and building one business process at\n  a time.\n- [Four-Step Dimensional Design Process](https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/four-4-step-design-process/)\n  — business process, grain, dimensions and facts, in that order.\n- [Type 2: Add New Row](https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/type-2/)\n  — keeping the history of a changing attribute with a new row, a new surrogate\n  key and three extra columns.\n- [BigQuery overview](https://docs.cloud.google.com/bigquery/docs/introduction)\n  — Google's managed warehouse, with storage and compute kept apart and data\n  stored by column.\n\n\u003chttps://engineer.company/glossary/data-warehouse/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"The database a business builds to answer questions rather than to run: the back room and the front room, history kept on purpose, and four ways it goes wrong.","formats":{"gemtext":"https://engineer.company/glossary/data-warehouse/index.gmi","gopher":"https://engineer.company/glossary/data-warehouse/gophermap","html":"https://engineer.company/glossary/data-warehouse/","json":"https://engineer.company/glossary/data-warehouse/index.json","markdown":"https://engineer.company/glossary/data-warehouse/index.md","pdf":"https://engineer.company/glossary/data-warehouse/index.pdf"},"kind":"page","language":"en","title":"What is a data warehouse?","url":"https://engineer.company/glossary/data-warehouse/","version":1}