{"canonical_url":"https://engineer.company/glossary/etl/","content_markdown":"# ETL\n\n## What it is\n\nETL, short for extract, transform, load, is the work of copying data out of the\nsystems that create it, putting it into the shape and meaning another system\nexpects, and writing it there, most often into a data warehouse. A close\nvariant, ELT, keeps the three steps but loads the raw data first and transforms\nit inside the target, and many teams write ETL/ELT to cover both.\n\nEach letter is a separate job. Extract reads from a source: a whole table, an\nexport file, an API, or only the rows that changed since the last run, found\nthrough an `updated_at` timestamp or through change data capture (CDC), which\nreads the database's own record of every change. Transform is where the data is\ncleaned and reconciled: text turned into dates and numbers, units and time zones\nmade consistent, duplicates removed, codes looked up against reference tables,\nand the result reshaped into the model the target uses, such as fact and\ndimension tables. Load writes the result, either as a bulk insert into an empty\ntable or as a merge that updates the rows already there and adds the new ones.\n\nThe order of the last two steps decides where the work runs. Classic ETL\ntransforms on a separate server before anything reaches the target, which made\nsense when warehouse storage and compute were scarce and costly. ELT lands the\nraw data in staging tables inside the target and runs the transformation there\nin SQL. The raw copy stays available, so when a rule changes the transformation\ncan be run again over history without extracting it a second time. The price is\nthat everything the source sent, including columns nobody asked for, now lives\nin the warehouse and has to be governed there.\n\nPostgreSQL covers each step with tools of its own. `COPY` moves rows between a\ntable and a file in text, CSV or binary format and is the fastest way in.\nReading a file on the server needs superuser rights or the\n`pg_read_server_files` role, while the psql command `\\copy` reads the file on\nthe client instead. `INSERT ... ON CONFLICT` and `MERGE` fold new rows into\nexisting ones. Logical decoding turns the write-ahead log into a stream of row\nchanges for CDC, and a PL/pgSQL procedure can keep the transformation next to\nthe data it changes.\n\n## Why it matters\n\nMost questions worth asking a business need data from more than one system.\nOrders sit in one database, payments in a provider's export, customer records in\na CRM. Until something moves them into one place under one set of rules, every\nanswer is assembled by hand, and the hand changes with the person doing it. ETL\nis that something, and a warehouse is only as correct as the loads that fill it.\n\nThe transform step is also where a business writes down what its data means.\nWhich rows count as a sale, which day a midnight payment belongs to, which\nrecord wins when two customers match: decided once in the load, the answer\napplies to every report downstream. Left to each report, the answers drift apart\nand two dashboards show two totals.\n\nHow a load extracts sets how it grows. A full reload copies everything every\ntime. It is simple and corrects itself, since yesterday's mistake is overwritten\ntonight, but its cost rises with the total size of the data. An incremental load\nmoves only what changed, so its cost rises with the rate of change instead. It\nhas to remember where the last run stopped, notice rows deleted at the source,\nand accept rows that arrive late with old timestamps. Most systems start with\nfull reloads and switch table by table once the night stops being long enough.\n\nThe ongoing cost is that loads are code that runs unattended, usually at night,\nagainst sources that change without notice. They need logging that says what ran\nand how many rows moved, alerts when a run stops, and one property above all: a\nrun repeated with the same input should leave the target exactly as one run\nwould. A load with that property can be retried after any failure without anyone\nchecking first.\n\n## What it looks like in practice\n\nFragmented pipelines were delaying business-critical reporting across several\nPostgreSQL development and production environments, and\n[600 ETL/ELT pipelines were written in PL/pgSQL](/portfolio/engineered-600-pl-pgsql-etl-elt-pipelines/)\nto replace them: ETL at the scale of hundreds of routine loads.\nPrimary and foreign key constraints guarded referential integrity during\ntransformations, unique and composite indexes supported the joins, and exception\nhandling with transactional control rolled a run back when it went wrong.\nIncremental loads used CDC and timestamp-based deltas, and execution logging\nand auditing supported tracking and debugging. The delivered work comprises\ntransformation logic, constraints, incremental loading and execution logging;\nthe record establishes no comparative benchmark or error-rate calculation.\n\nA migration is ETL that runs once. Moving a map application from Microsoft SQL\nServer to PostgreSQL with PostGIS meant carrying\n[over 50 million spatial records through ETL pipelines](/portfolio/improved-gis-map-performance-10x-on-postgresql/)\nwhose transformation steps converted every coordinate to the EPSG:4326 reference\nsystem. Once the records had landed, with GiST and SP-GiST indexes and a tuned\nconfiguration on the new side, spatial queries that had averaged 2.5 seconds\ncame back in under 250 milliseconds, ten times faster, and backend CPU load\ndropped by 65%.\n\nAcross a network boundary,\n[six ETL/ELT pipelines](/portfolio/designed-and-ran-6-etl-elt-pipelines/)\nshow every letter of the name at work. Extraction meant scheduled downloads of\ncompressed archives from a server reachable only through an IPsec VPN, with Bash\nand Python scripts sorting the Parquet, CSV and .bak files inside by type and\ncolumn layout, and the .bak backups restored into a local MSSQL Server instance. The\ndata was loaded into staging schemas in MSSQL and PostgreSQL with bcp, psql or\nSSIS depending on the format, cleaned, normalised and enriched by PL/pgSQL and\nT-SQL procedures, then compressed with gzip and carried through a Google Cloud\nStorage bucket into Google BigQuery. Work that had taken at least three hours\nby hand ran in under 20 minutes end to end, and the sync moved from weekly to\ndaily.\n\n## Where it goes wrong\n\n**A retried run loads the same rows twice.** A job inserts with plain `INSERT`,\nstops halfway on a network error, and the scheduler runs it again from the\nstart. Rows that made it in the first time are now in twice, and every total\nbuilt on them is too high. Give the target a key that identifies a row in\nbusiness terms and fold the batch in with `INSERT ... ON CONFLICT DO UPDATE`,\nwhich PostgreSQL documents as an atomic insert-or-update even under high\nconcurrency, or with `MERGE`. One catch: a single `ON CONFLICT DO UPDATE`\nstatement refuses to change the same row twice and raises a cardinality\nviolation, so a batch with repeated keys has to be reduced to one row per key\nfirst.\n\n```sql\nINSERT INTO meter_reading (meter_id, read_at, kwh)\nSELECT DISTINCT ON (meter_id, read_at) meter_id, read_at, kwh\nFROM staging_meter_reading\nORDER BY meter_id, read_at, received_at DESC\nON CONFLICT (meter_id, read_at)\nDO UPDATE SET kwh = EXCLUDED.kwh;\n```\n\n**A bulk load stops near the end and leaves the disk fuller than before.**\n`COPY FROM` writes rows into the table as it goes. When it stops on a bad row,\nthose rows stay behind in a deleted state, invisible to queries but still on\ndisk until `VACUUM` reclaims them, and a failure late in a large file can waste\na lot of space. PostgreSQL 17 added `ON_ERROR ignore`, which skips rows in text\nor CSV input whose values will not convert to their column's type, and\nPostgreSQL 18 added `REJECT_LIMIT` to cap how many it may skip. That covers type\nconversion only; a row that breaks a constraint still stops the command. Loading\ninto a staging table first, checking it there, and moving the good rows into the\nreal table in one transaction keeps a bad file away from the data people read.\n\n**Change data capture fills the disk.** Logical decoding reads changes through a\nreplication slot, and a slot keeps every piece of write-ahead log its consumer\nstill needs, whether or not a consumer is connected. When the reader stops, the\nlog piles up until the disk is full, and in extreme cases the server shuts down\nto prevent transaction ID wraparound. The PostgreSQL documentation is plain\nabout it: a slot that is no longer required should be dropped. After a crash, a\nslot can also send recent changes a second time, so the consumer has to tolerate\nduplicates, which is the same re-run property again.\n\n**The morning's first report is slow after a large load.** The planner chooses\nhow to run a query from statistics about each table, and a bulk load changes the\ndata under them. Autovacuum may refresh those statistics on its own, but not\nalways before the first report runs. Running `ANALYZE` as the last step of the\nload fixes it. For very large loads into an existing table, the PostgreSQL guide\nto populating a database adds that dropping indexes and foreign keys and\ncreating them again afterwards can pay off, because building them in bulk beats\nupdating them row by row. That trades away error checking while they are gone,\nso a unique index deserves a second thought before it goes.\n\n## Further reading\n\n- [COPY](https://www.postgresql.org/docs/current/sql-copy.html) — the bulk\n  loading command, its formats and options, and what a failed load leaves\n  behind.\n- [Populating a Database](https://www.postgresql.org/docs/current/populate.html)\n  — the PostgreSQL advice for large loads, from one transaction and `COPY` to\n  rebuilding indexes and running `ANALYZE` afterwards.\n- [MERGE](https://www.postgresql.org/docs/current/sql-merge.html) — insert,\n  update and delete in one statement, driven by how each source row matches the\n  target.\n- [INSERT](https://www.postgresql.org/docs/current/sql-insert.html) — the\n  `ON CONFLICT` clause, its atomic upsert and its rule against touching one row\n  twice.\n- [Logical Decoding Concepts](https://www.postgresql.org/docs/current/logicaldecoding-explanation.html)\n  — how changes are read from the write-ahead log and why an unused replication\n  slot has to go.\n\n\u003chttps://engineer.company/glossary/etl/\u003e\n","date_modified":"2026-11-08T05:31:18Z","description":"Extract, transform, load: the three steps that fill a warehouse, when loading first and transforming later wins, and four ways a nightly load goes wrong.","formats":{"gemtext":"https://engineer.company/glossary/etl/index.gmi","gopher":"https://engineer.company/glossary/etl/gophermap","html":"https://engineer.company/glossary/etl/","json":"https://engineer.company/glossary/etl/index.json","markdown":"https://engineer.company/glossary/etl/index.md","pdf":"https://engineer.company/glossary/etl/index.pdf"},"kind":"page","language":"en","title":"What is ETL?","url":"https://engineer.company/glossary/etl/","version":1}