Provenance and Lineage Tracking for Spatial Pipelines

A derived spatial layer is an assertion about the world produced by a chain of transformations, and it is only as trustworthy as its ability to name that chain. This guide is part of Geospatial Data Versioning Fundamentals & Architecture.

Prerequisites & Environment Setup

Before wiring lineage into a production pipeline, confirm each of the following:

Core Algorithmic Patterns

1. Reference by content, never by location

A lineage record that names paths is a record of where files happened to be. A record that names content hashes is a record of what was actually read. The distinction becomes concrete the first time someone overwrites an input in place: the path-based record still resolves, still looks valid, and now describes a run that can never be reproduced.

Every reference in the record — inputs, parameter files, grid-shift files, lookup tables — carries the hash of the bytes that were read. Paths may appear alongside for human convenience, but nothing downstream is allowed to resolve them.

2. Capture the environment, not just the code

Spatial results depend on library versions in ways that ordinary software rarely does. A PROJ data package update changes the transformation applied between two datums. A GDAL minor release changes a driver’s default precision. A GEOS upgrade changes which self-intersections make_valid resolves and how.

None of these are visible in the pipeline definition, and all of them change output bytes. The lineage record therefore captures the resolved environment:

{
  "toolchain": {
    "gdal": "3.9.2",
    "proj": "9.4.1",
    "proj_data": "1.19",
    "geos": "3.12.2",
    "geopandas": "1.0.1",
    "python": "3.11.9"
  }
}

The proj_data version is the field teams forget and the one that most often explains a coordinate shift nobody made.

3. Emit at the point of production

Lineage written by the stage that produced the output is correct by construction. Lineage written afterwards by a separate documentation task is an educated guess about what the pipeline did, and it degrades every time the pipeline changes and the documentation task does not.

This is why the record is declared as an output of the stage, alongside the data. If the record is missing, the stage failed — the same way a missing data file means the stage failed.

What a lineage record has to contain to be sufficient Three bands: the inputs band holds content hashes rather than paths, the process band holds resolved parameters and the producing commit, and the environment band holds the exact library versions including the PROJ data package. INPUTS content hashes grid files lookup tables PROCESS resolved parameters process steps commit ENVIRONMENT GDAL PROJ PROJ data GEOS consumed by executed under The bottom band is the one teams omit, and the one that explains a coordinate shift nobody made.

Production Workflow Implementation

Step 1 — Define the record

One JSON document per produced artifact, small enough to read and complete enough to rebuild from:

import hashlib
import json
import subprocess
from datetime import datetime, timezone
from pathlib import Path


def file_hash(path, chunk=1 << 20):
    """SHA-256 of a file's bytes, streamed so a 40 GB raster does not
    have to fit in memory."""
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        while block := fh.read(chunk):
            h.update(block)
    return h.hexdigest()


def toolchain():
    import geopandas, shapely
    from osgeo import gdal
    import pyproj
    return {
        "gdal": gdal.__version__,
        "proj": pyproj.proj_version_str,
        "proj_data": pyproj.datadir.get_data_dir().rsplit("/", 1)[-1],
        "geos": shapely.geos_version_string,
        "geopandas": geopandas.__version__,
    }


def lineage_record(output_path, inputs, parameters, process_steps, commit_sha):
    """Build the machine-readable lineage document for one artifact."""
    return {
        "schema": "gdv-lineage/1",
        "output": {
            "path": str(output_path),
            "sha256": file_hash(output_path),
            "bytes": Path(output_path).stat().st_size,
        },
        "inputs": [
            {"role": role, "path": str(p), "sha256": file_hash(p)}
            for role, p in inputs.items()
        ],
        "parameters": parameters,
        "process_steps": process_steps,
        "toolchain": toolchain(),
        "commit": commit_sha,
        "produced_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    }

produced_at is the one field that legitimately varies between two otherwise identical runs, which is why fingerprint comparison ignores it and compares the rest.

Step 2 — Declare the record as a pipeline output

In a DVC pipeline, the lineage file is an output of the stage that writes the data, so it is tracked, pushed and pulled with it:

# dvc.yaml
stages:
  reproject_parcels:
    cmd: >-
      python scripts/reproject.py
        --input data/raw/parcels.gpkg
        --output data/interim/parcels_3035.gpkg
        --lineage data/interim/parcels_3035.lineage.json
        --target-crs EPSG:3035
        --transformation EPSG:9691
    deps:
      - data/raw/parcels.gpkg
      - scripts/reproject.py
    params:
      - reproject.target_crs
      - reproject.transformation
    outs:
      - data/interim/parcels_3035.gpkg
      - data/interim/parcels_3035.lineage.json

Naming the transformation explicitly rather than letting PROJ pick one is what makes the reprojection defensible later; the reasoning is covered in CRS conflict resolution across branches.

Step 3 — Chain records into a lineage graph

Each record names its inputs by hash. Following those hashes backwards through the records of earlier stages reconstructs the full derivation without any central registry:

def trace(artifact_hash, records_by_output_hash, depth=0, seen=None):
    """Walk the lineage graph backwards from a published artifact."""
    seen = seen if seen is not None else set()
    if artifact_hash in seen:
        return                                   # a cycle means a broken record
    seen.add(artifact_hash)

    record = records_by_output_hash.get(artifact_hash)
    if record is None:
        print("  " * depth + f"[source] {artifact_hash[:12]}")
        return

    step = record["process_steps"][-1]["description"] if record["process_steps"] else "?"
    print("  " * depth + f"{artifact_hash[:12]}  ← {step}")
    for parent in record["inputs"]:
        trace(parent["sha256"], records_by_output_hash, depth + 1, seen)

A published basemap traced this way ends at a set of source hashes — the survey exports, the reference grids, the lookup tables — and every step between them is named. That output is what a data steward can actually review.

Step 4 — Export to ISO 19115 lineage

Catalogue consumers do not read the internal format. Generate the standard structure from it so the two cannot diverge:

def to_iso_lineage(record):
    """Map the machine record onto the ISO 19115 lineage structure."""
    return {
        "statement": (
            f"Derived by {record['process_steps'][-1]['description']} "
            f"using GDAL {record['toolchain']['gdal']} and "
            f"PROJ {record['toolchain']['proj']}."
        ),
        "processStep": [
            {
                "description": step["description"],
                "rationale": step.get("rationale", ""),
                "processingInformation": {
                    "identifier": record["commit"],
                    "softwareReference": record["toolchain"],
                    "parameters": record["parameters"],
                },
            }
            for step in record["process_steps"]
        ],
        "source": [
            {
                "description": inp["role"],
                "sourceCitation": {"identifier": inp["sha256"]},
            }
            for inp in record["inputs"]
        ],
    }

Generating rather than authoring this is the whole point: hand-written lineage statements describe the pipeline as it was when somebody last looked.

Step 5 — Verify by rebuilding

A lineage record is a claim, and the claim is testable:

# Provision from the record, rebuild, and compare everything but the timestamp
python scripts/rebuild_from_lineage.py data/interim/parcels_3035.lineage.json \
  --workdir /tmp/verify

python - <<'PY'
import json, hashlib
a = json.load(open("data/interim/parcels_3035.lineage.json"))
b = json.load(open("/tmp/verify/parcels_3035.lineage.json"))
for k in ("output", "inputs", "parameters", "toolchain"):
    assert a[k] == b[k], f"lineage mismatch in {k}"
print("rebuild reproduces the recorded artifact")
PY

Run this on a schedule against the most recent release, not only when someone asks. A record that stops verifying tells you a dependency moved, and it tells you before a consumer discovers it.

Following a published layer back to its sources A derivation graph across two lanes. A published basemap is traced backwards through a join and a reprojection to two source exports, each identified by content hash rather than by path, so the trace terminates at sources rather than at filenames. survey roads derived survey roads reproject join published Each edge is a hash match between one record's input and another's output — no central registry is involved.

Code Reliability Patterns

Fail the stage when a hash cannot be computed. A lineage record with a null hash is worse than no record, because it passes a presence check. If an input cannot be read, the stage has not run correctly.

Record the parameters that were resolved, not the ones that were requested. A parameter file that says simplify_tolerance: default records nothing. Resolve defaults to values before writing the record, so the number that was actually applied is the number that is stored.

Treat an environment change as a data change. When the toolchain moves, artifacts rebuilt with it get new hashes even if no source changed. That is correct and should be visible: a release note that says “rebuilt under PROJ 9.4.1” explains a diff that would otherwise look like corruption.

Keep the record small. A lineage document that grows into a log becomes something nobody reads and nobody validates. Inputs, parameters, toolchain, steps, output. Logs belong in the run’s job output.

Performance & Scale Considerations

Hashing dominates the cost of lineage on large artifacts, and it is unavoidable — but it is also usually already being done. If the artifacts are tracked by a content-addressed system, reuse its hash rather than recomputing one: reading a 40 GB raster twice to compute two different digests is pure waste.

For pipelines producing thousands of small artifacts, write one lineage record per stage covering all its outputs rather than one per file. The record then names a set of outputs with individual hashes, which keeps the file count manageable while preserving per-artifact addressability.

Where hashing genuinely hurts is on network storage. Compute the digest during the write that was happening anyway — stream the bytes through the hash as they are written — instead of reading the finished object back. On a 40 GB output over object storage this is the difference between one pass and two.

Troubleshooting & Failure Modes

Symptom Root Cause Fix
Rebuild produces different bytes with identical inputs An unpinned library default changed between runs Compare the toolchain blocks of the two records; pin the moved version and rebuild both
Coordinates shift with no code or data change PROJ data package updated on the runner Record and pin proj_data; re-run the affected stages deliberately and note it in the release
Lineage record references a hash nothing else knows An input was produced outside the pipeline Bring the input into the pipeline as a tracked source, or register it explicitly as an external source with its own record
Records exist but nobody can trace a published layer Records are not addressable by output hash Index the records by output hash at publication time; the trace walk depends on that index
ISO lineage disagrees with the internal record The standard export was hand-edited Regenerate the export; make the generated file an output of the pipeline so edits are overwritten
Verification passes but the artifact is wrong The record captures the run faithfully — the pipeline itself is at fault Lineage proves reproducibility, not correctness; pair it with the validation gates in CI/CD validation pipelines
A rebuild disagrees with the record — what changed? A decision ladder for diagnosing a failed reproduction: compare the toolchain blocks first, then the resolved parameters, then the input hashes, and only then conclude that the pipeline itself is non-deterministic. Do the two toolchain blocks differ? A library moved usually the PROJ data package yes no Do the resolved parameters differ? A default changed resolve defaults before recording them yes no Do the input hashes differ? An input was overwritten in place the path resolved to different bytes yes no The stage is non-deterministic unordered output or an embedded timestamp Checking in this order takes minutes; checking in any other order takes an afternoon.

FAQ

Is a pipeline definition in version control not already lineage?

It is half of it. The definition says what the pipeline would do; lineage says what it did — which input hashes it consumed, which parameter values were resolved, and which toolchain versions the runner had installed. A definition alone cannot distinguish two runs that produced different outputs because a library default moved between them, and that is the case lineage exists to explain.

How detailed should a lineage record be?

Detailed enough to rebuild the output, and no more. The test is mechanical: rebuild from the record alone on a clean machine and compare fingerprints. Whatever you had to look up elsewhere to make that work is a missing field. Anything beyond it is documentation, which is valuable but belongs somewhere else.

Where should lineage records be stored?

Beside the artifact, versioned with it, and referenced from its metadata. A record kept in a separate wiki or tracking database drifts from the data within a release or two. Writing it as a small JSON output of the producing stage means it travels with the data through every checkout, copy and publication — including into a release tag.

Does ISO 19115 lineage replace the machine-readable record?

No — it is the export format, not the source of truth. The standard lineage element is what catalogue consumers read and is deliberately narrative in places. Keep the precise machine record as the primary artifact and generate the conformant lineage from it, so the two can never disagree about what happened.

What about provenance for data a person edited by hand?

Hand edits are a legitimate process step and should be recorded as one: who, when, which features, and under which review. The manual review triggers for critical edits workflow already produces exactly this record; the lineage document should reference it rather than duplicating it.

Back to Geospatial Data Versioning Fundamentals & Architecture