{"canonical_url":"https://engineer.company/glossary/business-intelligence/","content_markdown":"# Business intelligence\n\n## What it is\n\nBusiness intelligence, usually shortened to BI, is the work of turning an\norganisation's own records into reports, dashboards and analyses that the people\nwho decide can act on. Used as an adjective it is often hyphenated, as in\nbusiness-intelligence software.\n\nThe phrase is older than the software. In October 1958 the IBM Journal of\nResearch and Development printed a paper by H. P. Luhn called \"A Business\nIntelligence System\". Luhn described a machine system that would take the\ndocuments an organisation receives or writes itself, abstract each one\nautomatically, describe it by a pattern of its words, and send it to the \"action\npoints\" whose profile of interests matched. An action point could be one person,\na group, a department or a whole division. He gave business a deliberately wide\nmeaning, any organised activity from science and commerce to law, government and\ndefence, and took intelligence to mean grasping how the facts in front of you\nrelate, well enough to steer action towards a goal. Having listed the\ntechniques, he named the whole: \"We call this a Business Intelligence System.\"\n\nLuhn's material was text. BI today mostly works on numbers in tables, and\nKimball Group's outline of a data warehouse and BI system shows the usual shape.\nA back room extracts data from the source systems, cleans it, makes it\nconsistent and loads it. A front room presents it, organised by business process\nand kept at the most detailed level, with aggregated summaries added for speed.\nOn top sit the BI applications: ad hoc queries, standard reports, dashboards and\nscorecards, and heavier analytic and data-mining applications.\n\nWhat joins the two meanings is Luhn's question: who needs to know this, and how\ndoes it reach them in a form they will actually read?\n\n## Why it matters\n\nEvery organisation already runs on numbers. The question is whether those\nnumbers agree. When sales counts an order on the day it is signed and finance\ncounts it on the day it is invoiced, the two departments bring two revenue\nfigures to the same meeting, and the meeting is spent on the difference instead\nof the decision. A BI system earns its cost by fixing one definition per\nmeasure, computing it in one place and showing the same figure to everyone who\nasks.\n\nLuhn's diagnosis from 1958 still fits most shared report folders. He saw\norganisations send information widely, just in case, and noted that the\nrecipients drowned in paper, which made it likely that nothing got through at\nall. A dashboard with forty charts does the same thing on a screen. His remedy\nwas selection. Each action point first received a short list identifying the\nchosen documents by number, name and author. It could then dial those numbers to\nget the abstracts, and accept the ones it wanted in full. Every acceptance\nupdated its profile, so the selection improved with use. A good report today\nworks the same way: it starts from a person and a decision, and shows the few\nfigures that decision depends on.\n\nThere is a practical side too. The database behind an application is built for\nits many small reads and writes. A question that reads a year of orders competes\nwith those writes when it runs in the same place, which is one reason BI data\nusually lives in a separate store loaded on a schedule. Keeping that store at\nthe most detailed level, as the Kimball outline does, decides which questions\ncan still be answered next year without going back to the sources.\n\n## What it looks like in practice\n\nA typical BI question: revenue for the third quarter of 2026, by region and\nproduct, with a subtotal for each region and a grand total. Given a table\n`sales` with one row per sale, holding `sold_on`, `region`, `product` and\n`amount`, standard SQL answers it in one query:\n\n```sql\nSELECT region,\n       product,\n       sum(amount)               AS revenue,\n       GROUPING(region, product) AS level\nFROM   sales\nWHERE  sold_on \u003e= DATE '2026-07-01'\nAND    sold_on \u003c  DATE '2026-10-01'\nGROUP  BY ROLLUP (region, product)\nORDER  BY GROUPING(region), region, GROUPING(product), product;\n```\n\nWith a handful of sample rows it returns:\n\n```text\n region | product | revenue | level\n--------+---------+---------+-------\n North  | Boots   |    1200 |     0\n North  | Jackets |     800 |     0\n North  |         |    2000 |     1\n South  | Boots   |     500 |     0\n South  | Jackets |    1500 |     0\n South  |         |    2000 |     1\n        |         |    4000 |     3\n```\n\n`ROLLUP (region, product)` is shorthand for\n`GROUPING SETS ((region, product), (region), ())`: the full list, every shorter\nprefix of it and finally the empty set, which adds up all rows. The PostgreSQL\ndocumentation describes it as the usual tool for hierarchical data, such as\nsalary by department, by division and for the whole company.\n`CUBE (region, product)` would group by every subset instead, and so add a total\nfor each product across all regions.\n\nIn a row's grouping set, a column that is not grouped comes back as NULL, which\nis why the subtotal rows have blank cells. `GROUPING(region, product)` says\nwhich columns those are. It returns a bit mask with one bit per argument, the\nrightmost argument in the lowest bit, and a bit is 1 when that column was not\ngrouped for the row. So 0 marks a detail row, 1 a region subtotal (product not\ngrouped) and 3 the grand total (neither grouped). A report reads `level` to\ndecide which rows to print in bold.\n\nThe company's\n[data analytics and BI dashboards service](/services/data-analytics-bi/) sets\nout its aim as dashboards and reports that show the handful of figures a\ndecision actually turns on, refreshed without anyone having to ask. A query like\nthis one is the layer under such a report: the totals are computed once, in the\ndatabase, and the dashboard only draws them.\n\n## Where it goes wrong\n\n**Two figures under one name.** Revenue, active customer and churn each have\nseveral reasonable definitions, and every report that computes its own picks\none. Write each measure down once, with what it counts, from which source and as\nof which date, and compute it in the database, for example as a view, rather\nthan in each report's formulas. Kimball calls the plain-language part business\nmetadata: what data is available, where it came from and what it means.\n\n**Everything sent to everyone.** A dashboard with dozens of charts and a mailing\nlist of the whole department reproduces the flood Luhn wanted to stop. Start\nfrom a named person and the decision they make, show the figures that decision\nneeds, and retire the reports that nobody has opened in a quarter.\n\n**Reports that slow the application down.** A heavy query on the primary\ndatabase of a live system competes with the customers using it. Run reporting on\na read replica or in a separate warehouse loaded on a schedule. Aggregates that\nare expensive and asked for often can go into a materialized view that is\nrefreshed after each load.\n\n**Totals built from totals.** The average order value of each region, averaged\nacross the regions, is not the company's average order value unless every region\nhas the same number of orders. A count of distinct customers per month cannot be\nadded up to a quarter, because a customer who bought in July and in August\ncounts twice. Keep the detailed rows and aggregate each level from them.\nGrouping sets do exactly that: every level of the query above is computed from\nthe sales rows, not from the level below it.\n\n**A blank that means two things.** Add one sale whose region is unknown, and the\nquery above returns a detail row with an empty region, a subtotal for it with\nboth cells empty, and the grand total with both cells empty too. With `CUBE` it\ngets worse: two rows read empty region and Boots, one for boots sold where the\nregion is unknown and one for boots across all regions. The PostgreSQL\ndocumentation warns that a NULL in the data cannot be told apart from the NULL\nof a grouped-away column. Test `GROUPING()`, never `IS NULL`, to find subtotal\nrows, and give missing values a visible label such as \"unknown\" when the data is\nloaded.\n\n**Yesterday's figures read as today's.** When a nightly load fails, the\ndashboard keeps showing the last good numbers, and nothing on screen says they\nare old. Print the time of the last successful load on every report, and have\nmonitoring raise an alert to a named person when a scheduled load does not\nfinish.\n\n## Further reading\n\n- [A Business Intelligence System](https://doi.org/10.1147/rd.24.0314) — H. P.\n  Luhn's paper in the IBM Journal of Research and Development, October 1958,\n  where the term comes from.\n- [Kimball Technical DW/BI System Architecture](https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/technical-dw-bi-system-architecture/)\n  — Kimball Group's outline of the back room, the presentation area, the BI\n  applications and the metadata.\n- [Table Expressions](https://www.postgresql.org/docs/current/queries-table-expressions.html)\n  — PostgreSQL's reference for `GROUPING SETS`, `ROLLUP` and `CUBE`, and how\n  NULL stands in for an ungrouped column.\n- [Aggregate Functions](https://www.postgresql.org/docs/current/functions-aggregate.html)\n  — the `GROUPING` function and its bit mask, with a worked example.\n\n\u003chttps://engineer.company/glossary/business-intelligence/\u003e\n","date_modified":"2026-10-30T02:51:31Z","description":"What business intelligence is: Luhn's 1958 idea, today's warehouse and reports, a SQL subtotal query with ROLLUP, and the ways BI goes wrong.","formats":{"gemtext":"https://engineer.company/glossary/business-intelligence/index.gmi","gopher":"https://engineer.company/glossary/business-intelligence/gophermap","html":"https://engineer.company/glossary/business-intelligence/","json":"https://engineer.company/glossary/business-intelligence/index.json","markdown":"https://engineer.company/glossary/business-intelligence/index.md","pdf":"https://engineer.company/glossary/business-intelligence/index.pdf"},"kind":"page","language":"en","title":"What is business intelligence?","url":"https://engineer.company/glossary/business-intelligence/","version":1}