{"canonical_url":"https://engineer.company/glossary/cte/","content_markdown":"# CTE\n\n## What it is\n\nA CTE, short for common table expression, is a query given a name in a WITH\nclause at the top of a statement, so that the rest of the statement can read its\nrows as though they came from a table. A statement can define several CTEs, each\none able to read the ones named before it, and the final query at the bottom\nputs them together. PostgreSQL's manual describes them as \"temporary tables that\nexist just for one query\", which is the right picture with one caveat: nothing\nis stored under that name, and the next statement cannot see it. Each CTE can be\na SELECT, or in PostgreSQL an INSERT, UPDATE, DELETE or MERGE, and the statement\nthe WITH clause is attached to can be any of those as well.\n\nA question such as \"which meters used more than three times their own usual amount on some day in the last 30 days\" splits naturally into steps:\n\n```sql\nWITH daily AS (\n  SELECT meter_id, read_at::date AS day, sum(kwh) AS kwh\n  FROM readings\n  WHERE read_at \u003e= current_date - 30\n  GROUP BY meter_id, read_at::date\n),\ntypical AS (\n  SELECT meter_id, avg(kwh) AS avg_kwh\n  FROM daily\n  GROUP BY meter_id\n)\nSELECT d.meter_id, d.day, d.kwh, t.avg_kwh\nFROM daily AS d\nJOIN typical AS t USING (meter_id)\nWHERE d.kwh \u003e 3 * t.avg_kwh\nORDER BY d.day, d.meter_id;\n```\n\nThe first step totals each meter's readings per day, the second averages those\ntotals per meter, and the last compares every day against its own meter's\naverage. The same answer can be written with nested subqueries, but the daily\ntotals would then appear twice in the text, once inside the other, and a reader\nwould have to work from the inside out. With WITH, the logic reads from top to\nbottom, and each step can be run on its own while it is being checked.\n\nAdding RECURSIVE lets a CTE read its own output, which is the part plain SQL has\nno other way to express. A recursive CTE has a starting query, then UNION or\nUNION ALL, then a query that refers back to the CTE. PostgreSQL runs the\nstarting query, keeps its rows in a working table, and keeps running the second\nquery against the newest rows until a round adds none. That is how a statement\nwalks a tree of unknown depth, such as the categories under one heading:\n\n```sql\nWITH RECURSIVE subtree AS (\n  SELECT id, parent_id, name, 1 AS depth\n  FROM categories\n  WHERE id = 42\n  UNION ALL\n  SELECT c.id, c.parent_id, c.name, s.depth + 1\n  FROM categories AS c\n  JOIN subtree AS s ON c.parent_id = s.id\n)\nSELECT id, name, depth FROM subtree ORDER BY depth, name;\n```\n\n## Why it matters\n\nThe first reason is legibility. An analytical query of a hundred lines that\nnests four levels deep is hard to review and harder to change. The same logic as\na chain of named steps gives a reviewer a sentence per step, gives an author a\nplace to put a comment, and lets anyone run one step alone to see whether its\nrows look right before trusting the next.\n\nThe second reason is how the database evaluates the steps, and here the details\nchanged over time. PostgreSQL normally evaluates a WITH query once per execution\nof the statement, even when the statement refers to it more than once, so an\nexpensive calculation needed in two places runs once. The cost of that guarantee\nis that the planner cannot push a filter from the outer query into a CTE that is\nreferenced several times, because the filter should apply to one use and not the\nothers. The CTE is computed in full and the rows are thrown away afterwards.\n\nUntil PostgreSQL 12, that was true of every CTE: each was evaluated before the\nrest of the query and never merged into it. Version 12 changed the default. A\nCTE that is not recursive, has no side effects and is referenced only once is\nnow folded into the outer query, so the planner optimises the two levels\ntogether and can carry the outer filter down to an index on the base table.\nMATERIALIZED forces the old behaviour, and NOT MATERIALIZED forces folding even\nfor a CTE used more than once, at the risk of running its work twice. SQLite\naccepts the same two keywords as non-binding hints and says it copied the idea\nfrom PostgreSQL. Other engines decide for themselves, and the plan is where to\ncheck.\n\nThe third reason is that a CTE can change data. A DELETE with RETURNING in a\nWITH clause can pass its output to an INSERT in the main statement, which moves\nrows from one table to another in a single statement that either happens in full\nor not at all. Data-modifying CTEs run exactly once and always to completion,\nwhether or not anything reads their output, which makes them a tool for doing\nseveral writes together rather than a way of describing a result.\n\n## What it looks like in practice\n\nThe work behind\n[Accelerated PostgreSQL performance by 10x](/portfolio/accelerated-postgresql-performance-by-10x/)\nwas a database for user accounts, tenants, locations and daily electricity\nreadings that slowed as the data grew. The fix came in three parts: targeted and\nGiST indexes, time-based partitioning of the readings table, and query rewrites.\nThe rewrites used CTEs and materialized views for frequently accessed data to\navoid full table scans, with EXPLAIN ANALYZE pointing at the joins and\nsubqueries worth restructuring. The tenfold improvement in response times and\nthe 40% drop in resource use came from the three parts together, not from CTEs\nalone. The pairing is worth noting, because the two do different jobs: a CTE\nnames an intermediate result for the length of one statement, while a\nmaterialized view stores a result that many statements can read until it is\nrefreshed.\n\nThe other side of CTEs, legibility, carried\n[the 500 electricity and GIS data analysis reports](/portfolio/released-500-electricity-and-gis-data-reports/).\nEvery report was written in pure SQL against PostgreSQL and BigQuery over\nsmart-meter data with coordinates, meter IDs and timestamped usage, and the\nscripts used CTEs together with window functions, subqueries and geospatial\njoins. Each report shipped with annotated SQL so that colleagues could reproduce\nand audit the analysis, plus notes on common data-quality issues such as missing\nGPS coordinates and corrupted meter values. SQL that reads step by step travels\nwell: the reports became a standard reference across departments and helped\nmake operational planning 10–15% more efficient. Named steps are what make SQL\nof that kind annotatable at all: a note can sit above each step and say what it\nremoves or assumes.\n\n## Where it goes wrong\n\n**A filter applied after all the work is done.** A CTE referenced twice, or\nmarked MATERIALIZED, is computed in full before the outer query filters it, and\nthe copy it produces has no indexes. A query that should touch a few rows reads\na whole table. Read the plan: when the CTE appears as its own step, with row\ncounts far above what the query returns, try NOT MATERIALIZED or restate the\nquery so the CTE is referenced once. Keep the materialised form where the CTE\ncalls an expensive function, so that the function runs once per row and not once\nper reference.\n\n**A recursive query that keeps going.** Data with a cycle, such as two\ncategories each listed as the other's parent, sends a recursive CTE with UNION\nALL round the loop for as long as the server lets it. UNION helps only when\nwhole rows repeat, which rarely holds once a depth column is added. Track the\npath of visited keys in an array and stop when a key reappears, or use the CYCLE\nclause that PostgreSQL 14 added. A LIMIT on the outer query works as a testing\ntrick, and the manual advises against relying on it in production. SQL Server\ncaps recursion at 100 levels by default and cancels the statement beyond that,\nso the same bug stops instead of running on.\n\n**Changes the rest of the statement does not see.** A data-modifying CTE and the\nmain query run concurrently on the same snapshot, so a main query that reads the\ntarget table directly sees the rows as they were before the change. The only\nchannel between them is RETURNING. Two parts of one statement that modify the\nsame row are worse: only one modification takes effect, and which one is hard to\npredict. Read changed rows from the CTE's RETURNING output, and keep each row to\na single writer per statement.\n\n**A plan that moved with an upgrade.** Code written for PostgreSQL 11 or earlier\nsometimes relied on a CTE being evaluated first, as a deliberate barrier against\na plan the author did not want. After an upgrade to 12 or later, the same CTE is\nfolded into the outer query and the planner picks something new, usually faster\nand occasionally much slower. Compare EXPLAIN output for the slow statements\nbefore and after the upgrade, and add MATERIALIZED where the barrier was\nintended.\n\n## Further reading\n\n- [7.8. WITH Queries (Common Table Expressions)](https://www.postgresql.org/docs/current/queries-with.html)\n  — the manual's chapter on CTEs: recursion, cycle detection, materialisation\n  and data-modifying statements.\n- [E.23. Release 12](https://www.postgresql.org/docs/12/release-12.html)\n  — the release that started folding single-use CTEs into the outer query.\n- [E.25. Release 14](https://www.postgresql.org/docs/14/release-14.html)\n  — the release that added the SQL-standard SEARCH and CYCLE clauses.\n- [The WITH Clause](https://www.sqlite.org/lang_with.html)\n  — SQLite's version, including its materialisation hints and where it\n  borrowed them from.\n- [WITH common_table_expression (Transact-SQL)](https://learn.microsoft.com/en-us/sql/t-sql/queries/with-common-table-expression-transact-sql)\n  — SQL Server's rules for CTEs, including the MAXRECURSION limit on\n  recursion.\n\n\u003chttps://engineer.company/glossary/cte/\u003e\n","date_modified":"2026-11-08T02:22:23Z","description":"What a CTE is in SQL, when PostgreSQL folds one into the outer query or computes it once, how recursive WITH works, and the traps it sets.","formats":{"gemtext":"https://engineer.company/glossary/cte/index.gmi","gopher":"https://engineer.company/glossary/cte/gophermap","html":"https://engineer.company/glossary/cte/","json":"https://engineer.company/glossary/cte/index.json","markdown":"https://engineer.company/glossary/cte/index.md","pdf":"https://engineer.company/glossary/cte/index.pdf"},"kind":"page","language":"en","title":"What is a CTE?","url":"https://engineer.company/glossary/cte/","version":1}