{"canonical_url":"https://engineer.company/glossary/pipeline/","content_markdown":"# Pipeline\n\n## What it is\n\nA pipeline, in full a data pipeline, is a set of jobs that run in a fixed order,\neach taking what the job before it produced, so that raw records from the\nsystems that create them end up cleaned, combined and stored where reports,\ndashboards or other programs can read them. The name comes from the Unix shell,\nwhere `|` feeds one command's output into the next. Most pipelines run\nunattended, on a schedule or whenever a new file arrives.\n\nFew pipelines are a straight line. A job that loads the day's sales may need\nboth the currency rates and the customer list first, while those two can run\nside by side. The jobs and the \"must finish first\" arrows between them form a\ndirected acyclic graph: every arrow points one way, and no job can end up\nwaiting on itself. Apache Airflow, an open-source platform for scheduling and\nmonitoring such workflows, calls each graph a Dag, written that way rather than\nDAG because its meaning in Airflow has grown past the bare mathematical\nstructure.\n\nA batch pipeline wakes on a schedule or an event, processes a bounded slice of\ndata, such as one day of meter readings, and stops. A streaming pipeline keeps\nrunning and passes each event on as it arrives. Airflow describes itself as a\nplatform for orchestrating batch workflows, best suited to work with a clear\nstart and end that runs on a schedule.\n\nThe simplest pipeline is a crontab line that starts a script at two in the\nmorning. That holds until several scripts must run in order, one of them breaks,\nand someone has to work out which ran and what to start again. An orchestrator\ntakes over that bookkeeping: it holds the graph, starts each job once the jobs\nupstream of it have succeeded, retries the ones that break, keeps a log per job\nand per run, and records which periods of data each run covered. In Airflow\nevery scheduled run covers a data interval. A daily run for 1 January starts\njust after midnight on the 2nd, once the day is over, and its jobs can read\nbounds such as `data_interval_start` to know exactly which slice of data is\ntheirs.\n\n## Why it matters\n\nA pipeline turns a procedure into code. The order of the jobs, the checks\nbetween them, what happens when one breaks and how often everything runs move\nout of one person's memory into files that can be reviewed, versioned and tested\nlike any other software. Airflow defines its workflows entirely in Python, an\napproach it calls workflows as code, and names version control and testing among\nthe benefits.\n\nA fault also gets an address. When the whole process is one long script, a crash\nat minute forty means starting again from minute zero. Split into jobs with\nrecorded state, a broken run shows which job stopped and why, and only that job\nand the ones after it need to run again. Airflow lets an operator rerun only the\nfailed tasks, or backfill runs for an earlier period, which is how a fix made on\nWednesday reaches Monday's data.\n\nRepetition becomes safe only when each job is written for it. Airflow's\nbest-practice guide asks each task to behave like a database transaction that\nnever leaves partial results behind and to \"produce the same outcome on every\nre-run\". A job built that way can be retried by the scheduler at three in the\nmorning, rerun after a fix or replayed for last month, and nobody has to inspect\nthe target first.\n\nThe price is software to own. An orchestrator is one more service with its own\ndatabase, upgrades and permissions, and every pipeline in it is code that needs\nreview and tests. For two scripts on one schedule, cron and a log file can be\nthe honest answer. The case for an orchestrator grows with the number of jobs\nthat depend on each other, the number of people who need to see what ran, and\nthe cost of a morning report built on a load that stopped halfway.\n\n## What it looks like in practice\n\nRunning many pipelines the same way was the problem on one platform, where\n[twenty GIS data pipelines and the application-data ETL](/portfolio/automated-delivery-of-20-gis-data-pipelines/)\nwere delivered and watched by hand, with a standing risk that one would stop\nquietly and leave the data downstream wrong before anyone noticed. All of them\ncame under automated delivery, with scheduling, logging and failure handling\nstandardised so that every pipeline behaved alike and a broken one showed, and\nwere folded into the existing infrastructure automation and reporting. All\ntwenty then ran automatically and predictably, with current data and nobody\nwalking it through by hand.\n\nA pipeline can also build something whole in one command. At one documented\nstage, a [database-driven CV and portfolio generator](/portfolio/built-a-database-driven-cv-generator/)\ndrew every document from one SQLite database of 23 tables, assembled by a\n19-step pipeline that ran from an empty file to a complete database. The steps\nwere ordered and each was written to be safe to repeat, so the pipeline could\nbe run against an existing database without duplicating a row. Forty-one Python\nmodules, 10,580 lines in all, then rendered six output formats from that one\nsource, with only two runtime dependencies. These counts describe the\nimplementation at that documented stage.\n\nInside a single job, speed matters as much as order. An\n[hourly electricity aggregation pipeline](/portfolio/a-month-of-meter-readings-in-180-milliseconds/)\ntakes JSONL data reported every hour, every 15 minutes or at irregular times,\naligns it to strict hourly intervals, sums the production mix, adds imports and\nsubtracts exports, and produces one row per hour per zone. It was built three\nways, in Python, in JQ and in SQL on PostgreSQL, and each version was\nbenchmarked on one day and thirty days of data. The Python version was the\nfastest, at 34 ms for one day and 180 ms for thirty, and its code was written so\nthat it could be extended to more zones or integrated into an ETL pipeline.\n\n## Where it goes wrong\n\n**A re-run gives a different answer.** A daily job appends its totals with\n`INSERT`, breaks halfway, and the scheduler retries it, so the rows written\nbefore the break are now there twice. A job that selects everything since the\ncurrent time minus a day reads a different slice on the retry than on the first\nattempt, and a job that reads the latest version of an input picks up whatever\nchanged in between. Airflow's best-practice guide names all three: replace\n`INSERT` with an upsert, read and write a specific partition keyed on the run's\n`data_interval_start`, and keep `now()` out of any critical computation. For an\nhourly summary table in PostgreSQL, one plain way is to replace the whole\ninterval inside a single transaction.\n\n```sql\nBEGIN;\nDELETE FROM hourly_load\nWHERE hour_start \u003e= :'start' AND hour_start \u003c :'end';\nINSERT INTO hourly_load (zone, hour_start, mwh)\nSELECT zone, date_trunc('hour', read_at), sum(mwh)\nFROM reading\nWHERE read_at \u003e= :'start' AND read_at \u003c :'end'\nGROUP BY zone, date_trunc('hour', read_at);\nCOMMIT;\n```\n\nRun as `psql -v ON_ERROR_STOP=1 -v start=... -v end=... -f hourly_load.sql`,\nwith the start and end of the run's data interval passed in by the orchestrator,\nthe job deletes and rewrites exactly that interval's rows, so a second run\nleaves the table as the first one did. The `:'start'` form is psql's own syntax\nfor inserting a variable quoted as an SQL literal.\n\n**A job finishes green and the data is wrong.** An orchestrator sees whether a\njob's process exited cleanly, and that is all it sees. psql is a sharp example:\nby default it carries on after an error in a script and returns 0 to the shell,\nso a step that runs `psql -f` reports success even when a statement inside it\nbroke. With `ON_ERROR_STOP` set, psql stops at the first error and exits with\ncode 3, so the step fails where it should. A clean exit still only proves the\nprogram ran, which is why Airflow's guide suggests self-checks: a downstream\ntask that confirms the partition was written and that the data looks right\nbefore anything reads it. For a run that hangs rather than breaks, Deadline\nAlerts, added in Airflow 3.1 to replace the SLA feature removed in 3.0, trigger\na callback, such as an email, once a run passes a time threshold.\n\n**Switching a pipeline back on starts a flood of runs.** With `catchup=True`,\nthe scheduler creates a run for every data interval it skipped since the start\ndate, and the Airflow documentation adds that catchup is also triggered when a\nDag is turned off for a while and then re-enabled. A daily pipeline paused for a\nmonth can come back with thirty runs queued against the same source. The\ndefault, `catchup_by_default=False`, makes the scheduler create a run only for\nthe latest interval, and the documentation says a Dag whose jobs read the\ncurrent time instead of their interval should keep catchup off. History is\nbetter filled on purpose with `airflow backfill create`, which takes a date\nrange and a limit on parallel runs through `--max-active-runs`.\n\n**Heavy code at the top of a Dag file slows the scheduler.** Airflow parses\nevery Dag file again and again, with at least `min_file_process_interval`\nseconds between parses, so code at the top level of the file, outside any task,\nruns far more often than the pipeline itself. A database query, a network call\nor a slow import there costs time on every parse, and the best-practice guide\nshows an example in which one such call makes a Dag take an additional 1000\nseconds to parse. Keep the top level to declaring tasks and their order, and\nmove expensive imports and calls inside the task functions that need them.\n\n## Further reading\n\n- [Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/index.html)\n  — the project's own overview: workflows as Python code, batch orchestration,\n  backfills and reruns of only the failed tasks.\n- [Best Practices](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)\n  — tasks treated as transactions, the same outcome on every re-run, top-level\n  code and self-checks.\n- [Dags](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html)\n  — how a Dag declares its schedule, tasks, dependencies and default arguments\n  such as retries, and where Deadline Alerts fit.\n- [Dag Runs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html)\n  — data intervals, catchup, backfill, and clearing tasks so they run again.\n- [Tasks](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html)\n  — dependencies between tasks, timeouts, retry policies and the move from SLAs\n  to Deadline Alerts.\n- [psql](https://www.postgresql.org/docs/current/app-psql.html) — variables and\n  their quoting, `ON_ERROR_STOP`, and the exit codes an orchestrator reads.\n\n\u003chttps://engineer.company/glossary/pipeline/\u003e\n","date_modified":"2026-11-09T21:50:25Z","description":"How a data pipeline orders, schedules and repeats its jobs, what an orchestrator adds to a crontab, and four ways a pipeline run goes wrong.","formats":{"gemtext":"https://engineer.company/glossary/pipeline/index.gmi","gopher":"https://engineer.company/glossary/pipeline/gophermap","html":"https://engineer.company/glossary/pipeline/","json":"https://engineer.company/glossary/pipeline/index.json","markdown":"https://engineer.company/glossary/pipeline/index.md","pdf":"https://engineer.company/glossary/pipeline/index.pdf"},"kind":"page","language":"en","title":"What is a pipeline?","url":"https://engineer.company/glossary/pipeline/","version":1}