{"canonical_url":"https://engineer.company/glossary/data-lineage/","content_markdown":"# Data lineage\n\n## What it is\n\nData lineage records how data reaches a destination through its sources and\ntransformations. A revenue figure on a dashboard may come from an order\ndataset, a table of refunds and a rule for converting currencies. Lineage\ngives the reader a path from the figure back to those inputs. It can also show\nwhich reports depend on an input before its owner changes a column or delivery\nschedule. The scope ranges from a table-level diagram to a recorded run with\nthe exact partitions and fields it read and wrote.\n\nOpenLineage models a dataset, a named job that transforms or moves data, and a\nrun, meaning a particular execution of that job. A run can report which\ndatasets it read and produced, and attach details about the environment,\nschema or source versions. Those event records give a team a way to tell\nwhich job was responsible for a particular transition. A static view of\nthe source code can also show possible dependencies. Both are useful when\nthe view makes clear how it obtained its evidence.\n\nChoose the granularity to fit the question. At dataset level an analyst can\nsee that a report depends on two tables. At column level a reviewer can trace\nthe origin of an email address or a currency field through selected expressions.\nAt execution level an operator can see which version of a transformation\nprocessed which batch and where it left its output. To retrace a disputed\nnumber, the time window, partition and rule version may matter as much as\nthe names of the tables.\n\n## Why it matters\n\nLineage makes a failure search smaller. When a daily figure suddenly falls,\nthe investigator can check whether the collection job brought in fewer\nrecords, a join dropped a group, or a report selected the wrong time window.\nA path narrows the first place to look. The next step is to confirm each\nsuspected break with observation, since a connected diagram alone cannot\nestablish that the data was correct or even current.\n\nKnowing the downstream users also helps plan a change. A developer renaming a\nstatus field can learn which reconciliation job and reports would be affected.\nThe owners can agree who will test the new name, where a temporary compatibility\nfield is needed and how long to keep it. Lineage enables this conversation\nwhile the change is still reversible. Let people see enough to understand\nthe risk while protecting data contents and permissions.\n\nWell maintained provenance supports confidence when different people use the\nsame label for different measures. A team can compare the dataset and\ntransformation definitions behind two numbers instead of treating the newer\nreport or the larger figure as authoritative. Lineage makes the question\nprecise enough to discuss with the owner. Correctness still needs a\ndefinition and a deliberate check of the result against that definition.\n\n## What it looks like in practice\n\nEngineer ApS describes operating [PostgreSQL, MS SQL and BigQuery warehouses](/portfolio/ran-100-postgresql-ms-sql-bigquery-warehouses/).\nWork across different warehouse engines makes source boundaries and scheduled\nprocessing relevant. The example below is a separate, fictional exercise for\nrecording evidence about a reporting flow.\n\nImagine a daily report that counts orders ready to ship. Orders arrive from\nthe storefront. A different feed delivers dispatch statuses. The report's\nquery groups by shipping date and chooses the newest status event for each\norder reference. Create a lineage entry for each input and the output, and name\nthe job that joins them. Record the job invocation identifier and its start\nand end times. Preserve the input partitions and the deployed revision that\nselects the newest status. If a run retries, record the final accepted output\nand keep the failed attempt visible as its own execution.\n\nNow trace an order that appears on a Tuesday report. Use the report date\nand order reference through approved queries. Follow the reported run to\nthe chosen dispatch record. Check the source event time against the shipping\ncutoff and verify how tied events are resolved. If the status feed was late,\nmake that visible in the description of the report and in the incident\ntimeline. When the cutoff changes, use the stored dependency information to\nask the affected owners to verify both an ordinary day and a late delivery.\n\nA small implementation can begin with one critical report and a hand-reviewed\nset of edges. Ask an analyst to trace a row and an operator to trace a failed\nrun. Note the point where either person reaches a guess. Add the missing\nidentifier, version or dataset in that order. Automatic collection has more\nvalue once the useful evidence and the limits of collection are explicit.\n\nKeep the status event time and the time the input became available. Record the publication cutoff too, so a pre-cutoff event that arrived late can be explained.\n\n## Where it goes wrong\n\nA diagram can look comprehensive when it has silently missed a manual export,\na stored procedure or a spreadsheet someone downloads and rewrites. Count\nwhat has been instrumented and state how the remaining path is documented.\nGive manually recorded edges a source and a date. A connected process that has\nchanged since review needs fresh evidence before someone relies on it.\n\nNames that change without a stable namespace can make two datasets appear to\nbe the same. Give datasets identifiers that include their environment and\nstorage context. An archive and a production table may have identical short\nnames. Group them only when the relationship is known. Keep enough history to\nexplain which identifiers were used before a move.\n\nFinally, protect the metadata itself. Table names, dependency patterns or\noperational notes can reveal information about sensitive work. Keep query\nstrings and example records out of general-purpose event fields when readers\nhave different permissions. Test important edges by making one known safe\nchange and checking the output and the path recorded for that run. A new input value can change the result while the dependency stays the same. Expect a different edge when the job actually changes its source. That supports a practical claim about the path you actually observed.\n\n## Further reading\n\n- [OpenLineage object model](https://openlineage.io/docs/spec/object-model/)\n- [W3C PROV-O provenance ontology](https://www.w3.org/TR/prov-o/)\n- [Google Cloud documentation on data lineage](https://docs.cloud.google.com/knowledge-catalog/docs/about-data-lineage)\n\n\u003chttps://engineer.company/glossary/data-lineage/\u003e\n","date_modified":"2026-11-19T19:18:49Z","description":"Trace a reported number through its datasets, runs and transformations, and use lineage evidence to assess the impact of a change.","formats":{"gemtext":"https://engineer.company/glossary/data-lineage/index.gmi","gopher":"https://engineer.company/glossary/data-lineage/gophermap","html":"https://engineer.company/glossary/data-lineage/","json":"https://engineer.company/glossary/data-lineage/index.json","markdown":"https://engineer.company/glossary/data-lineage/index.md","pdf":"https://engineer.company/glossary/data-lineage/index.pdf"},"kind":"page","language":"en","title":"What is data lineage?","url":"https://engineer.company/glossary/data-lineage/","version":1}