Integrating PDAL Pipelines into a Versioning Workflow

Bind a PDAL pipeline JSON to its exact input LAZ by hashing both, so that the read-filter-reclassify-tile transform is byte-reproducible and its outputs are safe to commit. Part of Point-Cloud Versioning and Branching Strategies.

Concept & Context

The tiling-and-hashing scheme that makes point-cloud versioning tractable is only as trustworthy as the process that produces the tiles. If the same source cloud can yield different tiles on two machines — because someone bumped a filter parameter, swapped a reclassification ruleset, or ran an older PDAL build — then the tile hashes diverge for reasons that have nothing to do with the data, and the version history becomes noise. Reproducibility has to start one step upstream, at the transform itself.

PDAL treats processing as a declarative pipeline: a JSON document listing a reader, a chain of filters, and a writer, each with explicit parameters. Because the pipeline is data, it can be hashed, committed, and replayed exactly like any other artifact. The discipline this page describes is to pin two hashes together — the pipeline JSON and the input LAZ — so that a run is fully determined by (pipeline_hash, input_hash). Given the same pair, any operator reproduces the same tiles, which then flow into the per-tile hashing and acquisition branching described in the parent guide. It is the point-cloud counterpart to the input pinning that large file handling in DVC for GIS applies to whole datasets.

Core Pipeline Steps

  1. Hash the input. Compute a SHA-256 of the source .laz and record it. This is the anchor the whole run is pinned to.
  2. Author the pipeline JSON. Write a reader → filters → tiled-writer pipeline with every parameter explicit — no defaults left implicit, no environment-dependent values.
  3. Pin the pipeline. Hash the canonicalized pipeline JSON and commit both the JSON and its hash to Git, so the transform itself is version-controlled.
  4. Run through a driver. Execute via a Python driver that verifies the input hash first, runs PDAL, and captures the stage metadata and logs.
  5. Commit the outputs. Push the tiles to the payload store and commit a run record binding input hash, pipeline hash, and output tile hashes.
A PDAL pipeline is a reader, filters, and a writer A chain of four stages: the reader opens the LAZ tile, the filter chain crops, reprojects and reclassifies, the writer emits a canonically ordered tile, and the hash of that tile becomes the version record. Reader LAZ tile in points Filter chain crop, reproject, classify points Writer canonical point order bytes Hash the version record Canonical ordering in the writer is what makes the hash stable across two identical runs.

Working Implementation

The pipeline below reads a LAZ, drops noise and low-confidence returns, reclassifies ground with a progressive morphological filter, and writes a fixed-capacity tile set. Every parameter is explicit so the JSON is a complete, self-describing transform.

{
  "pipeline": [
    {
      "type": "readers.las",
      "filename": "input/acquisition_2026_07_10.laz",
      "tag": "src"
    },
    {
      "type": "filters.range",
      "limits": "Classification![7:7],ReturnNumber[1:5]",
      "tag": "denoise"
    },
    {
      "type": "filters.assign",
      "value": "Classification = 0",
      "tag": "reset_class"
    },
    {
      "type": "filters.smrf",
      "scalar": 1.2,
      "slope": 0.2,
      "threshold": 0.45,
      "window": 16.0,
      "tag": "ground"
    },
    {
      "type": "filters.chipper",
      "capacity": 3000000,
      "tag": "tile"
    },
    {
      "type": "writers.las",
      "filename": "output/tiles/tile_#.laz",
      "compression": "laszip",
      "forward": "all",
      "a_srs": "EPSG:32632"
    }
  ]
}

The Python driver pins the run. It refuses to execute unless the input LAZ matches the recorded hash, hashes the pipeline JSON itself, runs PDAL, and writes a run record that binds inputs to outputs for the commit.

#!/usr/bin/env python3
"""Pinned PDAL driver: verify input hash, run pipeline, record provenance."""
import hashlib
import json
import sys
from pathlib import Path

import pdal


def sha256_file(path: str) -> str:
    """Stream a file through SHA-256 without loading it fully into memory."""
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()


def canonical_pipeline_hash(pipeline_json: str) -> str:
    """Hash the pipeline with keys sorted so formatting never affects the hash."""
    obj = json.loads(pipeline_json)
    canonical = json.dumps(obj, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()


def run_pinned(pipeline_path: str, expected_input_hash: str) -> dict:
    pipeline_json = Path(pipeline_path).read_text()
    spec = json.loads(pipeline_json)

    # Locate the reader's input file (first stage with a filename)
    input_path = next(s["filename"] for s in spec["pipeline"]
                      if isinstance(s, dict) and s.get("type", "").startswith("readers"))

    # Gate 1: input must match the pinned hash before we run anything
    actual_input_hash = sha256_file(input_path)
    if actual_input_hash != expected_input_hash:
        sys.exit(f"ABORT: input hash mismatch for {input_path}\n"
                 f"  expected {expected_input_hash}\n  got      {actual_input_hash}")

    # Execute PDAL and capture provenance
    pipeline = pdal.Pipeline(pipeline_json)
    point_count = pipeline.execute()
    metadata = pipeline.metadata          # per-stage CRS, bounds, counts
    log = pipeline.log                    # human-readable run log

    # Hash every output tile for the manifest downstream
    out_dir = Path("output/tiles")
    tile_hashes = {p.name: sha256_file(str(p))
                   for p in sorted(out_dir.glob("tile_*.laz"))}

    run_record = {
        "input_path": input_path,
        "input_hash": actual_input_hash,
        "pipeline_hash": canonical_pipeline_hash(pipeline_json),
        "points_processed": point_count,
        "tile_hashes": tile_hashes,
        "pdal_metadata": json.loads(metadata) if isinstance(metadata, str) else metadata,
    }
    Path("output/run_record.json").write_text(json.dumps(run_record, indent=2))
    return run_record


if __name__ == "__main__":
    # Usage: driver.py pipeline.json <expected_input_sha256>
    record = run_pinned(sys.argv[1], sys.argv[2])
    print(f"Processed {record['points_processed']} points into "
          f"{len(record['tile_hashes'])} tiles")
    print(f"pipeline_hash={record['pipeline_hash'][:12]}…")

Run it from the CLI, passing the input hash you pinned when the acquisition first landed:

# Record the input hash once, at ingest
sha256sum input/acquisition_2026_07_10.laz

# Every subsequent run is gated on that exact hash
python driver.py pipeline.json 4e8f...c17a

# Push tiles to the payload store, commit the pins + record
dvc add output/tiles
git add pipeline.json output/run_record.json output/tiles.dvc
git commit -m "acq 2026-07-10: SMRF ground reclass, 3M-pt tiles"
Wiring the pipeline into the repository Five steps: keep the pipeline definition in version control beside the data, pin the PDAL version, declare inputs and outputs as pipeline stages, run over tiles in parallel, and commit only the tiles whose hashes changed. 1 Version the pipeline definition the JSON is part of the data's provenance 2 Pin the toolchain a filter's default changing rewrites everything 3 Declare inputs and outputs so a re-run knows what is stale 4 Run per tile, in parallel tiles are independent by construction 5 Commit changed tiles only the manifest carries the rest The pipeline JSON is data provenance — a result without it is a number nobody can reproduce.

Validation & Output Verification

Reproducibility is worthless unless you can prove it, so verify the run before committing. First, confirm the pins: the run_record.json must carry both the input hash and the pipeline hash, and re-running canonical_pipeline_hash on the committed JSON must reproduce the recorded pipeline hash. Second, sanity-check the counts — PDAL’s metadata reports points in and points out per stage; assert that the tiled output point total matches the reader count minus whatever filters.range dropped.

# Re-derive the pipeline hash and compare to the committed record
python -c "import json,hashlib; \
o=json.load(open('pipeline.json')); \
c=json.dumps(o,sort_keys=True,separators=(',',':')); \
print(hashlib.sha256(c.encode()).hexdigest())"

# Confirm every tile listed in the run record exists and re-hashes correctly
python -c "import json,hashlib; \
r=json.load(open('output/run_record.json')); \
[print(n, 'OK' if hashlib.sha256(open(f'output/tiles/{n}','rb').read()).hexdigest()==h else 'FAIL') \
for n,h in r['tile_hashes'].items()]"

Finally, do the strongest check available: run the pinned pipeline twice on the same input and confirm the tile hashes are identical between runs. If they diverge, a non-deterministic parameter or PDAL version difference has leaked in, and the run is not yet reproducible.

Failure Modes

  • Symptom: Tile hashes differ between two runs of the same pinned pipeline. Root cause: A filter with order-dependent or threaded non-determinism, or a different PDAL/laz-perf build. Fix: Pin the PDAL version in the environment lock file and confirm filters like filters.smrf produce stable output; hash the sorted tile contents rather than raw bytes if writer ordering varies.
  • Symptom: Driver aborts with an input hash mismatch on a file you believe is unchanged. Root cause: The source LAZ was re-exported or re-flown, changing its bytes even though the survey looks the same. Fix: Treat it as a genuinely new input — record a new input hash and a new commit rather than forcing the old pin.
  • Symptom: Output point count is far below the reader count. Root cause: filters.range limits are too aggressive, silently discarding valid returns. Fix: Inspect PDAL metadata per-stage counts, widen the limits expression, and re-run; never let a filter silently drop the majority of a cloud.
  • Symptom: Tiles reassemble into the wrong CRS or a shifted position. Root cause: Missing a_srs on the writer or forward: "all" omitted, so header CRS/scale/offset were lost. Fix: Set forward: "all" and an explicit a_srs so every tile carries the source spatial reference.
Three reasons every tile looks changed A grid of three causes of total churn in a point-cloud pipeline: an unpinned PDAL version whose filter defaults moved, a writer left free to reorder points, and a compression setting that changed the bytes without changing a single point. Root cause Fix Every tile rewritten after an upgrade a filter default changed between versions pin PDAL and record the version in the manifest Identical input, different hash point order not canonicalised sort by the declared key before writing Bytes differ, points identical compression level changed hash the points, not the container The third row is worth building for: hashing content rather than bytes survives every encoder change.

Back to Point-Cloud Versioning and Branching Strategies