{"canonical_url":"https://engineer.company/glossary/analytics/","content_markdown":"# Data analytics\n\n## What it is\n\nTurning recorded facts into answers someone can act on is the job of data\nanalytics, often just called analytics. In the big data framework published by\nNIST, the US standards institute, analytics means working through data\nmethodically, reshaping it where needed, to bring out patterns, links between\ndata points and trends over time, and to try to predict what happens next. NIST\nplaces that step inside a wider _data analytics life cycle_ that turns raw data\ninto knowledge an organisation can use: collecting, preparing, analysing,\nvisualising and giving access.\n\nIt is a kind of data processing, set apart from the transactional kind by what\nit reads: an application writes one order at a time, while an analytical query\nreads millions of rows to return a handful. It also differs from data\nengineering, which builds the pipelines and storage that analytics reads from.\n\nThe word covers quite different work:\n\n- **Reporting**: the same figures delivered on a schedule, as a PDF or an email.\n- **Dashboards**: those figures kept live in a business intelligence tool such\n  as Power BI.\n- **Ad-hoc analysis**: a SQL query written for a question nobody asked before,\n  often against a warehouse such as BigQuery.\n- **Geospatial analysis**: the same work over map data in a GIS, asking where as\n  well as how much.\n- **Analytical applications**: a model or a statistical method behind a form,\n  for people who write no code.\n- **Web and product analytics**: counting the visitors to a website or the users\n  of a product, and what they do there.\n\nThe last kind carries a privacy cost, and it is the one meant when this site's\nprivacy, cookies, credits and security pages say “no analytics”: no analytics\ntag, no tracking pixel, no cookie and no account with an analytics company. The\nsite still measures itself. Its web server keeps an access log, and a program\nturns it into daily counts and a monthly ranking of pages read, referring sites\nand crawlers, kept as numbers and never joined into any one visitor's path.\n\n## Why it matters\n\nTwo choices settle most of what analytics will cost: where the queries run, and\nhow data about people is collected.\n\nAnalysis on the transactional database is the cheapest start: the data is\ncurrent and nothing is copied, but every report competes with the application\nfor memory and disk. A read replica moves that load off the primary for the\nprice of a second server. A warehouse such as BigQuery keeps storage apart from\ncomputing, and on its on-demand pricing bills each query by the bytes it reads,\nwhich rewards narrow queries, and it needs a pipeline to copy the data in.\n\nLock-in hides in the definitions. A BI tool keeps its measures in its own\nformula language, and a year later the meaning of an active customer lives in\none file only that tool opens. Definitions kept as SQL in the repository survive\na change of tool.\n\nWeb and product analytics adds a legal question. The usual hosted tool works\nthrough a tag: a script in every page that sets a cookie or a similar identifier\nin the browser and reports each page view to the vendor. Article 5(3) of the\nEU's ePrivacy Directive makes consent the condition for putting information on a\nvisitor's device or reading it back, with two exceptions: storage or access whose\nsole purpose is carrying a communication over a network, and storage or access\nstrictly necessary to deliver a service the visitor explicitly requested. Once\nevents can be tied to a person, even through a cookie identifier or an IP\naddress, the GDPR applies too: a lawful basis, a stated purpose, a retention\nperiod. And the vendor is another company, now holding a record of what your\nvisitors did.\n\nSo the real choice is among a hosted tool that is quick to add and comes with\ndashboards, a first-party collection you build and run yourself, and counts from\nlogs your server already writes, which answer fewer questions and carry far\nless.\n\n## What it looks like in practice\n\nFor a professional networking platform, the company built\n[product analytics inside PostgreSQL](/portfolio/built-first-party-analytics-inside-postgresql/)\nin place of a third-party tag: a schema of 7 tables and 47 functions, event\ntables partitioned with 22 partition definitions that prune themselves on a\nschedule, and visitor identity pseudonymised behind a rotating salt. The client\nsends nothing until the visitor has consented, and funnels are computed in the\ndatabase next to the data, which NIST calls data locality. The page names the\nprice: the rotating salt gives up long-horizon cohort analysis, and no free\ndashboards come with it.\n\nOn the reporting side, a\n[Python reporting system](/portfolio/developed-a-data-analytics-reporting-system/)\npulled the data, ran the analysis and delivered the same PDF report on a\npredictable schedule to the people making commercial decisions. The page credits\nthat reporting with raising quarterly software revenue by 400%.\n\nThis site is the third example. Its [privacy page](/privacy/) sets out the\nmeasurement: an access log that zeroes the final part of each IP address before\nwriting a line, the referring site kept as a name only, and lines that live\nabout thirty days. A program reads the log into a daily tally and a monthly\nranking, both kept indefinitely as counts with nothing that could join them back\ninto one visit. With no tag, no cookie and no third party, the page shows no\nconsent banner and gives the reason as “nothing to consent to”. It is open about\nthe limit too: while a line exists, a rare browser on a small network narrows a\nvisitor down further than the site would like. The trade is fewer answers, and\nno funnels, for nothing stored on the reader's device and nothing sent to anyone\nelse.\n\n## Where it goes wrong\n\n### Two tools report two visitor counts\n\nThe hosted tool shows one number for last month and the server log twice as\nmany, each right about something different. A tag counts only browsers that ran\nits script and, behind a banner, only visitors who agreed; a log counts every\nrequest, crawlers included, until someone filters them out. Write each metric's\ndefinition down once, in SQL, with what it includes and excludes.\n\n### Reports slow the application down\n\nEvery Monday the application slows while the weekly report scans whole tables on\nthe transactional database. Move the report to a replica or a warehouse, or roll\nraw rows into a small summary table each night and run window functions over\nthat:\n\n```sql\nSELECT day,\n       signups,\n       sum(signups) OVER (ORDER BY day\n                          ROWS BETWEEN 6 PRECEDING AND CURRENT ROW)\n         AS last_7_days\nFROM daily_counts\nORDER BY day;\n```\n\n### The banner goes and the tracking stays\n\nA team swaps cookies for a “cookieless” product, takes the consent banner down\nand goes on following visitors. Article 5(3) concerns access to information on\nthe device, whatever the mechanism, and the EDPB's guidelines on its technical\nscope say that fingerprinting from HTTP headers, or tracking navigation by IP\naddress alone, can fall under it too, while recalling that the rule applying\ndoes not by itself mean consent is needed. Judge a tool by whether it links one\nrequest to the next, not by whether it sets a cookie.\n\n### Consent that regulators would reject\n\nA banner blocks the page until the visitor accepts, treats scrolling as\nagreement, or makes withdrawing harder than agreeing was. The EDPB's consent\nguidelines say a cookie wall leaves consent not freely given and that scrolling\nis never a clear affirmative act, and Article 7(3) of the GDPR makes withdrawing\nas easy as giving. Give refusing the same weight as accepting, and send nothing\nbefore the answer.\n\n### Pseudonymous events kept forever\n\nThe event table grows by millions of rows a month, and the hashed user IDs in it\nare treated as anonymous. Recital 26 of the GDPR treats pseudonymised data that\nextra information could attribute to someone as personal data, and recital 30\nnames IP addresses and cookie identifiers among the online identifiers whose\ntraces, combined with other information, may be used to profile and identify\nsomeone. Set the retention period before the first event arrives, partition by\ntime so an expired month is dropped whole, and keep long-term figures as\naggregates without identifiers.\n\n## Further reading\n\n- [NIST SP 1500-1r2, Big Data Interoperability Framework: Definitions](https://doi.org/10.6028/NIST.SP.1500-1r2)\n  — the definitions of analytics, the data analytics life cycle and data\n  locality that this article starts from.\n- [Directive 2002/58/EC, consolidated](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:02002L0058-20091219)\n  — Article 5(3), the consent rule for anything stored on or read from a\n  visitor's device, with its two exceptions.\n- [Regulation (EU) 2016/679, the GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)\n  — the definition of consent, Article 7 on withdrawing it, and recitals 26 and\n  30 on pseudonymised data and online identifiers.\n- [EDPB Guidelines 05/2020 on consent](https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-052020-consent-under-regulation-2016679_en)\n  — cookie walls, scrolling and withdrawal, each with worked examples.\n- [EDPB Guidelines 2/2023 on Article 5(3)](https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-22023-technical-scope-art-53-eprivacy-directive_en)\n  — why IP-based tracking, fingerprinting and tracking pixels can fall under the\n  same rule as cookies.\n- [PostgreSQL tutorial: Window Functions](https://www.postgresql.org/docs/current/tutorial-window.html)\n  — rankings and running totals over related rows, computed inside the database.\n\n\u003chttps://engineer.company/glossary/analytics/\u003e\n","date_modified":"2026-10-20T21:59:50Z","description":"Reports, dashboards, warehouse SQL and visitor counts: where data analytics should run, and what web analytics costs in privacy and consent.","formats":{"gemtext":"https://engineer.company/glossary/analytics/index.gmi","gopher":"https://engineer.company/glossary/analytics/gophermap","html":"https://engineer.company/glossary/analytics/","json":"https://engineer.company/glossary/analytics/index.json","markdown":"https://engineer.company/glossary/analytics/index.md","pdf":"https://engineer.company/glossary/analytics/index.pdf"},"kind":"page","language":"en","title":"What is data analytics?","url":"https://engineer.company/glossary/analytics/","version":1}