{"canonical_url":"https://engineer.company/glossary/partitioning/","content_markdown":"# Partitioning\n\n## What it is\n\nPartitioning, also called table partitioning, stores the rows of one logical\ntable in several ordinary tables, each holding the rows whose key falls inside\nits own bounds, while queries and writes go on using the single name they always\nused. PostgreSQL, which this page follows, calls its built-in form declarative\npartitioning: the table is declared as partitioned, with a method and a\npartition key, and every partition is created as a table of its own.\n\nThe partitioned table holds no rows itself. An insert is routed to the partition\nwhose bounds match the key, and an update that changes the key moves the row to\nthe partition where it now belongs. The partitions share the parent's columns\nbut can have their own indexes, storage settings and tablespace, and a partition\ncan itself be partitioned again.\n\nThere are three methods. Range partitioning gives each partition a span of\nvalues, usually of dates; the lower bound is included and the upper one is not,\nso with spans 1–10 and 10–20 the value 10 belongs to the second. List\npartitioning names the values each partition takes, such as a set of country\ncodes. Hash partitioning gives each partition a modulus and a remainder, and a\nrow goes wherever the hash of its key leaves that remainder, which spreads rows\nevenly when there is no natural range to cut along. A default partition can\ncatch the values that fit no other, for range and list partitioning only.\n\nPartitioning happens inside one database server. Splitting data across separate\nservers, each with its own copy of the schema, is sharding, which is a much\nlarger step. SQL Server and BigQuery have their own forms of partitioning, with\ntheir own syntax, and the ideas below carry over even where the commands differ.\n\n## Why it matters\n\nMost large tables grow at one end. Orders, readings, logs and events arrive\nevery day, stay for months or years, and most queries ask about the recent part.\nWithout partitioning, every query works against one table and one set of indexes\nthat keep getting larger.\n\nWith a partition key that matches the queries, the planner can prove that most\npartitions cannot hold a matching row and leave them out of the plan entirely.\nPostgreSQL calls this partition pruning. A report on last week reads last week's\npartition and nothing else. The documentation adds a less obvious gain: because\nthe partitions take over the job of the upper levels of an index tree, the parts\nof the indexes that are used most are more likely to fit in memory.\n\nRemoving old data changes most of all. Deleting a month of rows from a single\ntable is a long `DELETE` that leaves dead rows behind for `VACUUM` to clean up.\nWith a partition per month, the same retention is one `DROP TABLE` or one\n`DETACH PARTITION`, which the documentation calls far faster than a bulk\noperation and which avoids the `VACUUM` overhead entirely. A detached partition\nis still an ordinary table, so it can be backed up or summarised before it is\ndropped, and partitions that are rarely read can sit on cheaper, slower storage.\n\nThe gain has a price, and the documentation is direct about when it pays. Each\npartition adds work to planning, and the benefits are normally worthwhile only\nwhen the table would otherwise be very large, with a rule of thumb that the\ntable should be larger than the database server's physical memory. Partitioning\nalso leaves one thing untouched: a single row that many transactions fight over\nis still one row in one partition, behind one lock.\n\n## What it looks like in practice\n\nAn `orders` table split by month on `created_at` is the plainest case, in a note\non scaling a database before sharding it that spends a section on\n[partitioning the tables around a hot row](/notes/optimise-before-you-shard/).\nThe primary key is `(id, created_at)` because the key has to include the\npartition column. A query that names a date range reads only the partitions\ninside it, this month's indexes stay small enough to live in memory, and\nretention becomes a single `DETACH PARTITION` in place of a `DELETE` of millions\nof rows. The limit is just as plain: partitioning does not make the contended\nrow faster, and a query that does not name the key reads every partition.\n\nA PostgreSQL performance project for an application holding user accounts,\ntenants, geographical data and daily electricity consumption\n[split the electricity table into time ranges](/portfolio/accelerated-postgresql-performance-by-10x/)\nby day and month, and used hash partitioning for the geographical data. That\ncame together with GiST and composite indexes and rewritten queries, and the\ncombined result was query response times improved by 10x and resource\nutilization down by 40%.\n\nAt a far larger scale, partitioning was one technique among several for\n[at least 100 databases across PostgreSQL, SQL Server and BigQuery](/portfolio/ran-100-postgresql-ms-sql-bigquery-warehouses/),\nmuch of it geographic and time-series data. Time-series partitioning was part of\nthe schema design in PostgreSQL and SQL Server, and BigQuery used\ntime-partitioned and clustered tables. Alongside spatial indexes, materialized\nviews and caching, query response times for that data fell by 40–60%.\n\n## Where it goes wrong\n\nPartitioning moves some of the work from query time to design time, and most\nproblems come from a design decision made without the queries in view.\n\n**Inserts start failing on the first day of a new month.** No partition covers\nthe new range, and PostgreSQL refuses an insert into the parent that maps to no\nexisting partition. Create partitions ahead of time, from a scheduled job that\ngenerates the DDL, as the documentation suggests, and alert when the newest\npartition is close to its end:\n\n```sql\nCREATE TABLE orders_2026_11\n  PARTITION OF orders\n  FOR VALUES FROM ('2026-11-01')\n              TO ('2026-12-01');\n```\n\nA default partition stops the errors, but rows for every uncovered month then\nland together in that one table, and attaching a new partition later scans the\ndefault under an `ACCESS EXCLUSIVE` lock unless a `CHECK` constraint on the\ndefault already excludes the new range. Treat a non-empty default partition as\nan alarm.\n\n**Queries are slower after partitioning than before.** The common queries do not\nfilter on the partition key, so each one reads every partition and its indexes.\nA second cause is the number of partitions: the documentation says the planner\ncopes well with up to a few thousand, provided typical queries prune all but a\nfew, and that planning time and memory grow beyond that. Choose the key from the\n`WHERE` clauses the application already writes, check with `EXPLAIN` that only\nthe expected partitions appear, and keep the count in proportion. One partition\nper customer is a typical trap, since the count grows with the business; the\ndocumentation suggests hash partitioning with a reasonable number of partitions\ninstead.\n\n**A primary key or unique constraint will not create.** On a partitioned table,\na unique or primary key must include every partition key column, and the\npartition key must not contain expressions or function calls. Each partition's\nindex can only enforce uniqueness inside that partition. Add the key column to\nthe constraint, as the note's `orders` table does with `(id, created_at)`.\n\n**Building an index on the parent blocks writes for a long time.** A plain\n`CREATE INDEX` on the parent locks out writes until every partition is indexed,\nand the `CONCURRENTLY` option works per partition rather than on the parent.\nCreate the index with `ON ONLY` on the parent, which leaves it marked invalid,\nbuild the index on each partition with `CONCURRENTLY`, and attach each one with\n`ALTER INDEX ... ATTACH PARTITION`. Once every partition's index is attached,\nthe parent index becomes valid by itself.\n\n**An existing large table cannot be switched over in place.** PostgreSQL does\nnot turn a regular table into a partitioned one. Create the new partitioned\ntable and attach the old table as one of its partitions. Before attaching, add a\n`CHECK` constraint that matches the partition's bounds, so the attach does not\nscan the whole table while holding an `ACCESS EXCLUSIVE` lock on it.\n\n## Further reading\n\n- [Table Partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html)\n  — the PostgreSQL chapter: the three methods, pruning, maintenance, limitations\n  and best practices.\n- [CREATE TABLE](https://www.postgresql.org/docs/current/sql-createtable.html)\n  — `PARTITION BY`, `PARTITION OF`, partition bounds and the default partition.\n- [ALTER TABLE](https://www.postgresql.org/docs/current/sql-altertable.html)\n  — `ATTACH PARTITION` and `DETACH PARTITION`, with what each one locks.\n- [CREATE INDEX](https://www.postgresql.org/docs/current/sql-createindex.html)\n  — indexes on partitioned tables, `ON ONLY` and building `CONCURRENTLY`.\n\n\u003chttps://engineer.company/glossary/partitioning/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"Partitioning keeps one table name over many smaller tables cut by a key, so queries skip the parts they do not need and old data leaves in one statement.","formats":{"gemtext":"https://engineer.company/glossary/partitioning/index.gmi","gopher":"https://engineer.company/glossary/partitioning/gophermap","html":"https://engineer.company/glossary/partitioning/","json":"https://engineer.company/glossary/partitioning/index.json","markdown":"https://engineer.company/glossary/partitioning/index.md","pdf":"https://engineer.company/glossary/partitioning/index.pdf"},"kind":"page","language":"en","title":"What is partitioning?","url":"https://engineer.company/glossary/partitioning/","version":1}