{"canonical_url":"https://engineer.company/glossary/dataset/","content_markdown":"# Dataset\n\n## What it is\n\nA dataset, also written data set, is a bounded body of data that people handle\nas one thing: it has a name, a source, a period it covers and a shape its\nrecords share, and those four facts say where it starts and stops. A month of\nmeter readings in one file is a dataset; so is a nightly export of three related\ntables. The word says nothing about size or format, only that the data inside\nbelongs together.\n\nThe pages on this site use the word in that working sense. One speaks of a\nhistorical dataset in JSONL format and of 1-day and 30-day datasets used for\nbenchmarks, another of curated datasets and tables exported for loading\nelsewhere, a third of vast datasets generated by smart meters. Each time, the\nword draws a line around data before something is done to it.\n\nThe W3C's Data Catalog Vocabulary, DCAT, gives the word a formal frame. Version\n3 became a W3C Recommendation on 22 August 2024, and it defines a dataset as a\n\"collection of data, published or curated by a single agent, and available for\naccess or download in one or more serializations or formats\". The notion is\nbroad on purpose: numbers, text, imagery and sound all qualify. DCAT then\nseparates three things that everyday speech runs together:\n\n- **Dataset**: the data as an idea, such as hourly consumption per zone\n  for 2024.\n- **Distribution**: one accessible form of it, such as a downloadable CSV file;\n  one dataset may have several.\n- **Dataset series**: datasets published separately that share enough to be\n  grouped, such as one file per month.\n\nSo two files in different formats may be one dataset, and two monthly files\nunder one name may be two members of a series.\n\nThe word has a second, narrower meaning in Google BigQuery. There a dataset is a\ncontainer for tables, views and other resources, and it sits inside a project. A\ntable has to belong to a dataset, so a dataset is created before any data is\nloaded, and it is addressed as `project.dataset` in GoogleSQL and as\n`project:dataset` in the bq command-line tool. Context decides which meaning is\nin play.\n\n## Why it matters\n\nA boundary makes data comparable. Two sets of readings can be compared only when\nboth cover the same period, the same places and the same spacing between\nrecords, and naming a dataset is how those terms get fixed. DCAT gives the\nspacing its own property, temporal resolution, meaning the smallest gap between\nthe dataset's items, written as a duration such as `P1D` for one day. A dataset\nthat states its resolution tells the reader whether its rows are hours,\nquarter-hours or something irregular before the first sum.\n\nDescription has to travel with the data. DCAT lists what a distribution can say\nabout itself: a media type, a format, a compression format, a packaging format,\na byte size and a checksum, the last of which lets a receiver verify that the\ncontents have stayed unchanged. The specification's own example of a compression\nformat is gzip. A file that arrives with those facts can be checked and opened;\na file that arrives bare has to be guessed at.\n\nThe format of a distribution decides who can read it. Three are common:\n\n- **CSV.** RFC 4180, from 2005, records the layout most implementations seem to\n  follow: one record per line, an optional header line, double quotes around any\n  field that holds a comma, a line break or a double quote, and a double quote\n  inside a field written twice. Almost any tool opens it.\n- **JSON Lines.** One valid JSON value per line, UTF-8, with `\\n` as the line\n  separator, saved as `.jsonl` or, gzipped, `.jsonl.gz`. A final newline is\n  strongly recommended, not required. Programs read it one record at a time, and\n  Unix-style text tools and shell pipes handle it well.\n- **Parquet.** An open-source data file format that keeps the values of each\n  column together. It is built to store and retrieve data efficiently, and its\n  compression and encoding schemes let it handle complex data in bulk.\n\nHere is one record in the first two, where the comma inside the note forces\nquotes in the CSV:\n\n```text\nid,read_at,kwh,note\nM-17,2024-01-08T10:00:00Z,3.21,\"late, manual\"\n\n{\"id\":\"M-17\",\"read_at\":\"2024-01-08T10:00:00Z\",\"kwh\":3.21,\"note\":\"late, manual\"}\n```\n\n## What it looks like in practice\n\nShaping one dataset out of many sources was the task when electricity data\narrived with each source's own reporting frequency, from hourly intervals to\n15-minute intervals and in some cases irregular timestamps, which made a\ncoherent, comparable dataset hard to produce. The work\n[took raw event data from a historical JSONL dataset and turned it into hourly consumption](/portfolio/a-month-of-meter-readings-in-180-milliseconds/),\none row per hour and per zone, summing the production mix in each interval,\nadding imports and subtracting exports. Three variants were written, in Python,\nJQ and SQL. The Python pipeline parsed the JSONL files into data frames,\nresampled the time series to hourly intervals, and exported the results as CSV\nor loaded them into a lightweight SQLite database for inspection. Benchmarked on\n1-day and 30-day historical datasets, it was the fastest of the three, at 34 ms\nfor one day and 180 ms for thirty. The work between input and output is a change\nof grain, from each source's own spacing to one fixed row per hour and zone.\n\nMoving a dataset is a different job. A set of\n[six ETL/ELT pipelines carried data from a remote warehouse behind an IPsec VPN into Google BigQuery](/portfolio/designed-and-ran-6-etl-elt-pipelines/).\nThe data arrived as compressed archives containing Parquet, CSV and .bak files,\nand Bash and Python scripts classified each file by type and schema. Once\nprocedures had cleaned, normalised and enriched the rows, curated datasets and\ntables were exported to intermediate formats, compressed with gzip and\ntransferred to a Google Cloud Storage bucket, from which the bq CLI and Python\nscripts loaded them into BigQuery. Getting those datasets there by hand had\ntaken 3+ hours; automated, the trip took under 20 minutes, and the tables were\nrefreshed every day where they had been refreshed once a week.\n\nReading one is the third. Smart meters across multiple geographic regions\nproduced\n[vast datasets behind more than 500 analysis reports written in pure SQL](/portfolio/released-500-electricity-and-gis-data-reports/)\non PostgreSQL and BigQuery. The data held geolocation coordinates, meter IDs,\ntimestamped energy usage and environmental metadata. Each report carried\nannotated SQL so that colleagues could reproduce and audit it, and\ntroubleshooting notes recorded common data quality issues, such as missing GPS\ncoordinates or corrupted meter values, with recommended handling. Datasets that\ncolleagues can reproduce and audit are datasets planners will trust, and the\nwork contributed to a 10–15% gain in operational planning efficiency.\n\n## Where it goes wrong\n\nMost dataset trouble is a fact about the data that stayed in one person's head.\n\n**The grain goes unstated.** Hourly rows added to quarter-hour rows count some\nintervals four times over. Write the grain down in one sentence, such as one row\nper hour per zone in UTC, and align every source to it before totalling.\n\n**A gap reads as zero.** An empty field, a null and a 0 mean three different\nthings, and a total quietly treats a missing reading as nothing used. Missing\nGPS coordinates and corrupted meter values were common enough on the smart-meter\nproject to earn documented handling. Decide per column whether a gap is dropped,\nfilled or flagged, and say which in the dataset's notes.\n\n**The CSV dialect drifts.** RFC 4180 documents the common layout, and real files\nstray from it: another separator, a missing header, a field with a comma and its\nquotes left off. The RFC leaves the header optional, signalled only by a MIME\nparameter that a file copied to disk leaves behind. Check each new file's first\nline against the expected header before loading.\n\n**A copy forgets where it came from.** A curated extract loaded into a second\nsystem looks just like the original a month later. Record the source, the\nperiod, the extraction time and a checksum next to every copy so that a report\ncan say which extract it read.\n\n**A BigQuery dataset lands in the wrong place.** A dataset's location is fixed\nat creation; moving it means copying or recreating the dataset. Every table in\none query must sit in datasets in the same location, so choose the location\nbefore the first load.\n\n## Further reading\n\n- [Data Catalog Vocabulary (DCAT) - Version 3](https://www.w3.org/TR/vocab-dcat-3/)\n  — the W3C definitions of dataset, distribution and dataset series, with the\n  properties that describe each.\n- [BigQuery: Introduction to datasets](https://cloud.google.com/bigquery/docs/datasets-intro)\n  — the container sense of the word: projects, locations, naming and limits.\n- [JSON Lines](https://jsonlines.org/) — the three rules of the format and its\n  file-naming conventions.\n- [RFC 4180: Common Format and MIME Type for CSV Files](https://www.rfc-editor.org/rfc/rfc4180)\n  — the CSV layout most tools follow, its quoting rules and the text/csv type.\n- [Apache Parquet: Overview](https://parquet.apache.org/docs/overview/) — what\n  the column-oriented file format is and where its specification lives.\n\n\u003chttps://engineer.company/glossary/dataset/\u003e\n","date_modified":"2026-11-08T05:31:18Z","description":"What makes a body of data a dataset, how DCAT and BigQuery use the word, which file formats carry one, and where shared data loses its meaning.","formats":{"gemtext":"https://engineer.company/glossary/dataset/index.gmi","gopher":"https://engineer.company/glossary/dataset/gophermap","html":"https://engineer.company/glossary/dataset/","json":"https://engineer.company/glossary/dataset/index.json","markdown":"https://engineer.company/glossary/dataset/index.md","pdf":"https://engineer.company/glossary/dataset/index.pdf"},"kind":"page","language":"en","title":"What is a dataset?","url":"https://engineer.company/glossary/dataset/","version":1}