{"canonical_url":"https://engineer.company/glossary/bigquery/","content_markdown":"# BigQuery\n\n## What it is\n\nBigQuery is a managed SQL analytics service on Google Cloud: nobody provisions a server for\nit, tables are stored column by column, and each query is paid for either by the bytes it\nreads or out of compute capacity reserved in advance. It is also sold and searched for as\nGoogle BigQuery, and the SQL it speaks is GoogleSQL, a dialect that follows the ANSI\nstandard and adds functions of its own.\n\nThe design rests on keeping storage and compute apart. Tables live in Google's storage\nlayer, and a query borrows compute for as long as it runs. Because each column is stored\nseparately, a query that names three columns of a fifty-column table reads those three and\nleaves the rest on disk. Tuning therefore looks different from tuning a PostgreSQL server.\nInstead of designing B-tree indexes and sizing memory, the work is the shape of the table:\nit can be partitioned by a date or timestamp column, by ingestion time or by an integer\nrange, and clustered on up to four columns so that related rows share storage blocks.\n\nBigQuery belongs to the family of data warehouses, and the products it is usually weighed\nagainst are Snowflake and Amazon Redshift, plus the option of keeping analytics on the\ntransactional database a company already runs. It is built for queries that sweep millions\nor billions of rows and return a handful of totals. The single-row lookups and updates an\napplication makes on every request belong in a database such as PostgreSQL or SQL Server,\nand a common arrangement runs both, with a pipeline carrying data from one to the other.\n\n## Why it matters\n\nThe decision BigQuery puts in front of a team is how it wants to pay for analysis. The\nbill follows the queries people write, not the size of a machine somebody once chose, and\nthat cuts both ways.\n\nUnder on-demand pricing, every query is charged for the bytes it processes. That suits\nirregular work: a team that runs a few heavy reports a week pays for those reports and for\nnothing in between, with no server idling overnight. It also gives a careless query a\nprice. A `SELECT *` over a year of sensor readings costs the same whether the analyst\nwanted one row or all of them. Capacity pricing replaces this with slots, units of compute\nbought through reservations, which turns the bill into a figure known in advance and makes\nheavy, steady use cheaper to plan for. A project can move between the two models, so the\nfirst choice can be revisited once real usage is visible.\n\nStorage has rules of its own. A table or partition left unmodified for 90 consecutive days\nmoves to the long-term rate, 50 percent lower, without anyone moving it. Each dataset is\nbilled either on the logical, uncompressed size of its data or on the physical, compressed\nsize, and for data that compresses well the second model can be the cheaper one.\n\nThree things deserve weight before signing up. The location of a dataset, a single region\nor the US or EU multi-region, is set when the dataset is created and stays fixed, so it is\na compliance decision taken on the first day. The dialect, the `bq` command-line tool and\nthe client libraries are Google's own, so queries move to another warehouse with edits\nrather than unchanged; the data itself leaves easily, since tables export to CSV, Avro and\nParquet. And the hiring question is gentler than it looks: analysts who know SQL are\nproductive in days, while the rarer skill is reading a query in bytes as well as in rows.\n\nSo when is BigQuery the right call? When data arrives in large, mostly appended volumes\nsuch as events, logs or meter readings, when the questions asked of it change from week to\nweek, and when the team would rather not operate database servers for analysis. For data\nthat a single PostgreSQL server holds comfortably, a warehouse adds a second system, a\nsecond bill and a pipeline between the two. The case for it grows with the volume and with\nthe number of people asking questions.\n\n## What it looks like in practice\n\nAn estate of more than 100 databases, split between PostgreSQL, Microsoft SQL Server and\nGoogle BigQuery, held mostly geographic or time-series data: coordinates, sensor\nreadings, logs and real-time metrics. BigQuery handled the large-scale time-series data,\n[its tables time-partitioned and clustered](/portfolio/ran-100-postgresql-ms-sql-bigquery-warehouses/),\nwith compression and columnar storage applied to keep storage costs down and scans fast.\nPartitioned, clustered scans in BigQuery, together with indexing, materialized views and\ncaching across the estate, took 40–60% off the response times of GIS and time-series\nqueries.\n\nOn a geographical data pipeline that had grown slow, the SQL and the data models in\nPostgreSQL, MS SQL and Google Cloud BigQuery were analysed first, which surfaced weak\nindexing, inefficient queries and missing constraints. Each engine then got the tool it\nhas: B-tree and GiST indexes in PostgreSQL, filtered indexes in MS SQL and\n[clustering in BigQuery](/portfolio/made-a-geographical-data-pipeline-50x-faster/),\nalongside normalised models, partitioning and rewritten queries. The founder's achievement\nrecord reports that the pipeline as a whole ran 50x faster; it does not specify the benchmark\ndataset, hardware or comparison procedure.\n\nAnd on smart-meter data from several regions,\n[more than 500 analysis reports were written in pure SQL](/portfolio/released-500-electricity-and-gis-data-reports/)\nagainst PostgreSQL and BigQuery, with CTEs, window functions, subqueries and geospatial\njoins over meter IDs, timestamps, coordinates and environmental metadata. Each report\nshipped with its SQL annotated, so that a colleague could rerun and audit it, and with\nnotes on known data-quality problems such as missing GPS coordinates and corrupted meter\nvalues. The reports became a reference across departments and contributed to a 10–15%\nimprovement in operational planning efficiency.\n\nIn all three cases BigQuery worked next to PostgreSQL rather than in its place, and each\nengine was given the work it does best.\n\n## Where it goes wrong\n\n**A `LIMIT 10` query is billed for the whole table.** On a table without clustering,\n`LIMIT` trims the result, not the bytes read, and `SELECT *` reads every column there is.\nA quick look at the data becomes a full scan. Name the columns the question needs, and for\na glance at sample rows use the console's table preview or `bq head`, which cost nothing.\n\n**A partitioned table gets scanned end to end anyway.** Partitions are skipped only when\nthe filter on the partitioning column is a constant the engine can evaluate before\nreading. Comparing the column with a subquery's result, wrapping it in a function or doing\narithmetic on it sends the query through every partition. Two guards help. A table can\nrequire a partition filter, so a query without a usable one stops with an error instead of\nrunning. And a dry run reports the bytes a query would process without running it or\ncharging for it:\n\n```sh\nbq query --use_legacy_sql=false --dry_run \\\n  'SELECT meter_id, kwh\n   FROM energy.readings\n   WHERE reading_date = DATE \"2026-10-01\"'\n```\n\n**Clustering is added and the bytes read barely move.** Clustering sorts storage blocks by\nup to four columns in order, and the order matters: a table clustered by order date,\ncountry and status is not organised for a filter on country and status alone. Tables under\n64 MB gain little from it, and on a clustered table the bytes actually scanned are known\nonly after the query finishes. Order the clustering columns the way queries filter. Where\ncost has to be capped before anything runs, rely on partitioning, whose pruning a dry run\nalready accounts for, and set a maximum bytes billed so that a query over the limit stops\nbefore it is charged.\n\n**A query that joins two datasets is rejected.** Every dataset lives in one location, and\na query runs in one location, so every dataset it touches has to share it. Teams that let\neach project take its own default end up with tables that can only be joined after one of\nthem is copied across. Pick the location once, for compliance and for where the data comes\nfrom, and create every dataset there on purpose.\n\n## Further reading\n\n- [BigQuery overview](https://cloud.google.com/bigquery/docs/introduction)\n  — Google's own description of the service, its separate storage and compute layers and\n  what is managed for you.\n- [Introduction to partitioned tables](https://docs.cloud.google.com/bigquery/docs/partitioned-tables)\n  — the partitioning types, their granularity and their limits.\n- [Query partitioned tables](https://docs.cloud.google.com/bigquery/docs/querying-partitioned-tables)\n  — which filters prune partitions and which send a query through all of them.\n- [Introduction to clustered tables](https://docs.cloud.google.com/bigquery/docs/clustered-tables)\n  — how column order shapes block pruning, and what that means for cost estimates.\n- [Estimate and control costs](https://docs.cloud.google.com/bigquery/docs/best-practices-costs)\n  — dry runs, maximum bytes billed and the free ways to preview a table.\n- [BigQuery locations](https://docs.cloud.google.com/bigquery/docs/locations) — regions,\n  multi-regions and the rules for queries that touch more than one dataset.\n\n\u003chttps://engineer.company/glossary/bigquery/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"How BigQuery bills queries and storage, when it is worth more than analytics on your own database, and four mistakes that make one query costly.","formats":{"gemtext":"https://engineer.company/glossary/bigquery/index.gmi","gopher":"https://engineer.company/glossary/bigquery/gophermap","html":"https://engineer.company/glossary/bigquery/","json":"https://engineer.company/glossary/bigquery/index.json","markdown":"https://engineer.company/glossary/bigquery/index.md","pdf":"https://engineer.company/glossary/bigquery/index.pdf"},"kind":"page","language":"en","title":"What is BigQuery?","url":"https://engineer.company/glossary/bigquery/","version":1}