# Dataset ## What it is A dataset, also written data set, is a bounded body of data that people handle as one thing: it has a name, a source, a period it covers and a shape its records share, and those four facts say where it starts and stops. A month of meter readings in one file is a dataset; so is a nightly export of three related tables. The word says nothing about size or format, only that the data inside belongs together. The pages on this site use the word in that working sense. One speaks of a historical dataset in JSONL format and of 1-day and 30-day datasets used for benchmarks, another of curated datasets and tables exported for loading elsewhere, a third of vast datasets generated by smart meters. Each time, the word draws a line around data before something is done to it. The W3C's Data Catalog Vocabulary, DCAT, gives the word a formal frame. Version 3 became a W3C Recommendation on 22 August 2024, and it defines a dataset as a "collection of data, published or curated by a single agent, and available for access or download in one or more serializations or formats". The notion is broad on purpose: numbers, text, imagery and sound all qualify. DCAT then separates three things that everyday speech runs together: - **Dataset**: the data as an idea, such as hourly consumption per zone for 2024. - **Distribution**: one accessible form of it, such as a downloadable CSV file; one dataset may have several. - **Dataset series**: datasets published separately that share enough to be grouped, such as one file per month. So two files in different formats may be one dataset, and two monthly files under one name may be two members of a series. The word has a second, narrower meaning in Google BigQuery. There a dataset is a container for tables, views and other resources, and it sits inside a project. A table has to belong to a dataset, so a dataset is created before any data is loaded, and it is addressed as `project.dataset` in GoogleSQL and as `project:dataset` in the bq command-line tool. Context decides which meaning is in play. ## Why it matters A boundary makes data comparable. Two sets of readings can be compared only when both cover the same period, the same places and the same spacing between records, and naming a dataset is how those terms get fixed. DCAT gives the spacing its own property, temporal resolution, meaning the smallest gap between the dataset's items, written as a duration such as `P1D` for one day. A dataset that states its resolution tells the reader whether its rows are hours, quarter-hours or something irregular before the first sum. Description has to travel with the data. DCAT lists what a distribution can say about itself: a media type, a format, a compression format, a packaging format, a byte size and a checksum, the last of which lets a receiver verify that the contents have stayed unchanged. The specification's own example of a compression format is gzip. A file that arrives with those facts can be checked and opened; a file that arrives bare has to be guessed at. The format of a distribution decides who can read it. Three are common: - **CSV.** RFC 4180, from 2005, records the layout most implementations seem to follow: one record per line, an optional header line, double quotes around any field that holds a comma, a line break or a double quote, and a double quote inside a field written twice. Almost any tool opens it. - **JSON Lines.** One valid JSON value per line, UTF-8, each line ending in `\n`, saved as `.jsonl` or, gzipped, `.jsonl.gz`. Programs read it one record at a time, and Unix-style text tools and shell pipes handle it well. - **Parquet.** An open-source data file format that keeps the values of each column together. It is built to store and retrieve data efficiently, and its compression and encoding schemes let it handle complex data in bulk. Here is one record in the first two, where the comma inside the note forces quotes in the CSV: ```text id,read_at,kwh,note M-17,2024-01-08T10:00:00Z,3.21,"late, manual" {"id":"M-17","read_at":"2024-01-08T10:00:00Z","kwh":3.21,"note":"late, manual"} ``` ## What it looks like in practice Shaping one dataset out of many sources was the task when electricity data arrived with each source's own reporting frequency, from hourly intervals to 15-minute intervals and in some cases irregular timestamps, which made a coherent, comparable dataset hard to produce. The work [took raw event data from a historical JSONL dataset and turned it into hourly consumption](/portfolio/engineered-hourly-electricity-consumption-aggregation-pipeline-in-python-4/), one row per hour and per zone, summing the production mix in each interval, adding imports and subtracting exports. Three variants were written, in Python, JQ and SQL. The Python pipeline parsed the JSONL files into data frames, resampled the time series to hourly intervals, and exported the results as CSV or loaded them into a lightweight SQLite database for inspection. Benchmarked on 1-day and 30-day historical datasets, it was the fastest of the three, at 34 ms for one day and 180 ms for thirty. The work between input and output is a change of grain, from each source's own spacing to one fixed row per hour and zone. Moving a dataset is a different job. A set of [six ETL/ELT pipelines carried data from a remote warehouse behind an IPsec VPN into Google BigQuery](/portfolio/designed-implemented-and-administered-6-etl-elt-pipelines-8/). The data arrived as compressed archives containing Parquet, CSV and .bak files, and Bash and Python scripts classified each file by type and schema. Once procedures had cleaned, normalised and enriched the rows, curated datasets and tables were exported to intermediate formats, compressed with gzip and transferred to a Google Cloud Storage bucket, from which the bq CLI and Python scripts loaded them into BigQuery. Getting those datasets there by hand had taken 3+ hours; automated, the trip took under 20 minutes, and the tables were refreshed every day where they had been refreshed once a week. Reading one is the third. Smart meters across multiple geographic regions produced [vast datasets behind more than 500 analysis reports written in pure SQL](/portfolio/released-500-electricity-and-gis-data-analysis-reports-11/) on PostgreSQL and BigQuery. The data held geolocation coordinates, meter IDs, timestamped energy usage and environmental metadata. Each report carried annotated SQL so that colleagues could reproduce and audit it, and troubleshooting notes recorded common data quality issues, such as missing GPS coordinates or corrupted meter values, with recommended handling. Datasets that colleagues can reproduce and audit are datasets planners will trust, and the work contributed to a 10–15% gain in operational planning efficiency. ## Where it goes wrong Most dataset trouble is a fact about the data that stayed in one person's head. **The grain goes unstated.** Hourly rows added to quarter-hour rows count some intervals four times over. Write the grain down in one sentence, such as one row per hour per zone in UTC, and align every source to it before totalling. **A gap reads as zero.** An empty field, a null and a 0 mean three different things, and a total quietly treats a missing reading as nothing used. Missing GPS coordinates and corrupted meter values were common enough on the smart-meter project to earn documented handling. Decide per column whether a gap is dropped, filled or flagged, and say which in the dataset's notes. **The CSV dialect drifts.** RFC 4180 documents the common layout, and real files stray from it: another separator, a missing header, a field with a comma and its quotes left off. The RFC leaves the header optional, signalled only by a MIME parameter that a file copied to disk leaves behind. Check each new file's first line against the expected header before loading. **A copy forgets where it came from.** A curated extract loaded into a second system looks just like the original a month later. Record the source, the period, the extraction time and a checksum next to every copy so that a report can say which extract it read. **A BigQuery dataset lands in the wrong place.** A dataset's location is fixed at creation; moving it means copying or recreating the dataset. Every table in one query must sit in datasets in the same location, so choose the location before the first load. ## Further reading - [Data Catalog Vocabulary (DCAT) - Version 3](https://www.w3.org/TR/vocab-dcat-3/) — the W3C definitions of dataset, distribution and dataset series, with the properties that describe each. - [BigQuery: Introduction to datasets](https://cloud.google.com/bigquery/docs/datasets-intro) — the container sense of the word: projects, locations, naming and limits. - [JSON Lines](https://jsonlines.org/) — the three rules of the format and its file-naming conventions. - [RFC 4180: Common Format and MIME Type for CSV Files](https://www.rfc-editor.org/rfc/rfc4180) — the CSV layout most tools follow, its quoting rules and the text/csv type. - [Apache Parquet: Overview](https://parquet.apache.org/docs/overview/) — what the column-oriented file format is and where its specification lives.