Emitting ISO 19115 Lineage from a DVC Pipeline

The pipeline already knows every stage it ran, every hash it consumed and every parameter it resolved — so the lineage a catalogue publishes should be generated from that record rather than written from memory. This page is a focused companion to provenance and lineage tracking for spatial pipelines.

Concept & Context

ISO 19115 models lineage as three things: a narrative statement, an ordered set of LI_ProcessStep elements describing what was done, and a set of LI_Source elements describing what it was done to. That structure maps almost exactly onto a pipeline definition — stages are process steps and external inputs are sources — which is why generating it is straightforward once you decide to.

Most published lineage is a single sentence because it is written by a person at publication time, and a sentence is what a person can maintain. The consequence is a catalogue full of entries that say “derived from national survey data using standard processing”, which satisfies the schema and answers nothing. Generated lineage costs nothing per release and carries the hashes, versions and parameter values that make a dataset reproducible.

The generator reads the pipeline lock file rather than the pipeline definition. This matters: the definition describes intent, while the lock file records the resolved hashes of what actually ran. Reading the definition produces lineage for a pipeline that might have run.

Reading the definition, versus reading the lock file Two panels contrasting the two sources a lineage generator could read. The pipeline definition describes intent and contains ranges and defaults; the lock file records the resolved hashes and parameter values from the run that actually happened. THE DEFINITION Describes what the pipeline would do Holds parameter names, not resolved values Unchanged when an input changes Produces lineage for a run that may never have occurred THE LOCK FILE Records what the last successful run consumed Holds resolved hashes for deps and outs Changes whenever any upstream stage does Makes the lineage stage re-run at the right time Depend on the lock file and the lineage cannot describe a pipeline that did not run. This is also why the lineage stage declares the lock file as a dependency.

Core Algorithmic Pipeline

  1. Read dvc.lock, which holds, per stage, the resolved dependency hashes, parameter values and output hashes from the last successful run.
  2. Build the stage graph by matching each stage’s dependency hashes against other stages’ output hashes; unmatched dependencies are external sources.
  3. Emit one LI_ProcessStep per stage, in topological order, carrying the command, the resolved parameters and the toolchain versions.
  4. Emit one LI_Source per external input, citing its content hash as the identifier.
  5. Serialise to ISO 19139 XML and validate against the schema before it goes anywhere near a catalogue.
How a pipeline maps onto the standard's lineage elements A grid mapping each part of a pipeline record onto the ISO 19115 lineage element that carries it: stages become process steps, external inputs become sources, resolved parameters become processing information, and the toolchain goes into the statement. ISO element Carries Pipeline stage LI_ProcessStep the command and its position in the order External input LI_Source the content hash, cited as the identifier Resolved parameters processingInformation values as applied, not as requested Toolchain statement the versions that produced these bytes The last row is narrative in the standard, which is why it has to be generated rather than written.

Working Implementation

"""Generate ISO 19115-2 lineage from dvc.lock and the pipeline's toolchain record."""
from __future__ import annotations

import subprocess
from datetime import datetime, timezone
from xml.etree import ElementTree as ET

import yaml

GMD = "http://www.isotc211.org/2005/gmd"
GCO = "http://www.isotc211.org/2005/gco"
ET.register_namespace("gmd", GMD)
ET.register_namespace("gco", GCO)


def q(ns: str, tag: str) -> str:
    return f"{{{ns}}}{tag}"


def char_string(parent: ET.Element, tag: str, text: str) -> ET.Element:
    """<gmd:tag><gco:CharacterString>text</gco:CharacterString></gmd:tag>"""
    el = ET.SubElement(parent, q(GMD, tag))
    cs = ET.SubElement(el, q(GCO, "CharacterString"))
    cs.text = text
    return el


def load_lock(path: str = "dvc.lock") -> dict:
    with open(path, encoding="utf-8") as fh:
        return yaml.safe_load(fh)["stages"]


def classify(stages: dict) -> tuple[list[str], list[dict]]:
    """Split dependencies into internal (produced by a stage) and external sources."""
    produced = {
        out["md5"]: name
        for name, stage in stages.items()
        for out in stage.get("outs", [])
        if "md5" in out
    }
    order, sources = [], []
    for name, stage in stages.items():
        order.append(name)
        for dep in stage.get("deps", []):
            if dep.get("md5") and dep["md5"] not in produced:
                sources.append({"path": dep["path"], "md5": dep["md5"]})
    return order, sources


def toolchain_statement() -> str:
    import pyproj
    from osgeo import gdal
    return (f"GDAL {gdal.__version__}, PROJ {pyproj.proj_version_str}, "
            f"PROJ data {pyproj.datadir.get_data_dir().rsplit('/', 1)[-1]}")


def build_lineage(stages: dict, dataset_title: str) -> ET.Element:
    order, sources = classify(stages)
    commit = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()

    lineage = ET.Element(q(GMD, "LI_Lineage"))
    char_string(
        lineage, "statement",
        f"{dataset_title} produced by a versioned processing pipeline at commit "
        f"{commit[:12]} using {toolchain_statement()}. "
        f"{len(order)} process step(s) applied to {len(sources)} source dataset(s)."
    )

    for position, name in enumerate(order, start=1):
        stage = stages[name]
        step_wrap = ET.SubElement(lineage, q(GMD, "processStep"))
        step = ET.SubElement(step_wrap, q(GMD, "LI_ProcessStep"))
        char_string(step, "description", f"[{position}] {name}: {stage.get('cmd', '')}")

        params = stage.get("params", {})
        if params:
            flat = "; ".join(
                f"{file}:{key}={value}"
                for file, entries in params.items()
                for key, value in entries.items()
            )
            char_string(step, "rationale", f"parameters {flat}")

        date_wrap = ET.SubElement(step, q(GMD, "dateTime"))
        dt = ET.SubElement(date_wrap, q(GCO, "DateTime"))
        dt.text = datetime.now(timezone.utc).isoformat(timespec="seconds")

    for src in sources:
        src_wrap = ET.SubElement(lineage, q(GMD, "source"))
        li_source = ET.SubElement(src_wrap, q(GMD, "LI_Source"))
        char_string(li_source, "description",
                    f"{src['path']} (sha/md5 {src['md5']})")

    return lineage


if __name__ == "__main__":
    tree = ET.ElementTree(build_lineage(load_lock(), "National Parcel Basemap"))
    ET.indent(tree, space="  ")
    tree.write("metadata/lineage.xml", encoding="utf-8", xml_declaration=True)
    print("wrote metadata/lineage.xml")

Wire it in as a stage so the XML is an output of the pipeline rather than a manual step:

# dvc.yaml
stages:
  lineage:
    cmd: python scripts/emit_lineage.py
    deps:
      - dvc.lock
      - scripts/emit_lineage.py
    outs:
      - metadata/lineage.xml:
          cache: false        # small, readable, belongs in Git rather than the cache

Declaring dvc.lock as a dependency is what makes the stage re-run whenever any upstream stage produced something new — which is exactly when the lineage needs regenerating.

Validation & Output Verification

Validate the XML before a catalogue does it for you, because most catalogues reject quietly:

# Well-formed, and conformant against the ISO 19139 schema
xmllint --noout metadata/lineage.xml
xmllint --noout --schema schemas/gmd/gmd.xsd metadata/lineage.xml

# Every process step must name a stage that exists in the lock file
python - <<'PY'
import re, yaml
from xml.etree import ElementTree as ET
GMD = "{http://www.isotc211.org/2005/gmd}"
GCO = "{http://www.isotc211.org/2005/gco}"

stages = set(yaml.safe_load(open("dvc.lock"))["stages"])
tree = ET.parse("metadata/lineage.xml")
described = set()
for desc in tree.iter(f"{GMD}description"):
    text = desc.find(f"{GCO}CharacterString").text or ""
    m = re.match(r"\[\d+\] ([^:]+):", text)
    if m:
        described.add(m.group(1))

missing = stages - described
assert not missing, f"stages absent from lineage: {sorted(missing)}"
print(f"{len(described)} process step(s) cover every stage in dvc.lock")
PY

Then confirm the lineage actually changes when the pipeline does:

# Change a parameter, re-run, and the XML must differ
git stash list >/dev/null
yq -i '.reproject.target_crs = "EPSG:3857"' params.yaml
dvc repro lineage
git diff --stat metadata/lineage.xml     # expected: a change
git checkout params.yaml && dvc repro lineage

A lineage file that does not change when a parameter changes is reading the definition rather than the lock file, and is describing a pipeline that may never have run.

Four checks before a catalogue sees the XML Four validation steps: well-formedness, conformance against the profile schema the catalogue publishes, coverage of every stage in the lock file, and a responsiveness check confirming the lineage changes when a parameter does. 1 Well-formed XML cheapest failure to find 2 Conformant to the profile the catalogue's schema, not the base one 3 Covers every stage a stage absent from lineage is a silent gap 4 Responds to a change change a parameter; the XML must differ Catalogues reject quietly, so the second step is the difference between published and apparently published.

Failure Modes

  • Every input appears as an external source — symptom: LI_Source entries for intermediate files the pipeline itself produced. Root cause: dependency hashes compared against the wrong field, so no internal match was found. Fix: match dependency md5 against each stage’s output md5, as classify() does.

  • The XML validates but the catalogue rejects it — symptom: a silent ingestion failure. Root cause: the catalogue enforces a national profile stricter than base ISO 19139. Fix: validate against the profile schema the catalogue publishes, not the base schema.

  • Lineage stops updating — symptom: published lineage describes a pipeline from three releases ago. Root cause: the lineage stage does not depend on dvc.lock, so it never re-runs. Fix: declare the lock file as a dependency.

  • Parameter values appear as default — symptom: recorded parameters that convey nothing. Root cause: the parameter file stores a sentinel and resolution happens in code. Fix: resolve defaults before writing them, so the recorded value is the value that was applied.

Back to Provenance and Lineage Tracking for Spatial Pipelines