Integrating PDAL Pipelines into a Versioning Workflow

Bind a PDAL pipeline JSON to its exact input LAZ by hashing both, so that the read-filter-reclassify-tile transform is byte-reproducible and its outputs are safe to commit. Part of Point-Cloud Versioning and Branching Strategies.

Concept & Context

The tiling-and-hashing scheme that makes point-cloud versioning tractable is only as trustworthy as the process that produces the tiles. If the same source cloud can yield different tiles on two machines — because someone bumped a filter parameter, swapped a reclassification ruleset, or ran an older PDAL build — then the tile hashes diverge for reasons that have nothing to do with the data, and the version history becomes noise. Reproducibility has to start one step upstream, at the transform itself.

PDAL treats processing as a declarative pipeline: a JSON document listing a reader, a chain of filters, and a writer, each with explicit parameters. Because the pipeline is data, it can be hashed, committed, and replayed exactly like any other artifact. The discipline this page describes is to pin two hashes together — the pipeline JSON and the input LAZ — so that a run is fully determined by (pipeline_hash, input_hash). Given the same pair, any operator reproduces the same tiles, which then flow into the per-tile hashing and acquisition branching described in the parent guide. It is the point-cloud counterpart to the input pinning that large file handling in DVC for GIS applies to whole datasets.

Core Pipeline Steps

  1. Hash the input. Compute a SHA-256 of the source .laz and record it. This is the anchor the whole run is pinned to.
  2. Author the pipeline JSON. Write a reader → filters → tiled-writer pipeline with every parameter explicit — no defaults left implicit, no environment-dependent values.
  3. Pin the pipeline. Hash the canonicalized pipeline JSON and commit both the JSON and its hash to Git, so the transform itself is version-controlled.
  4. Run through a driver. Execute via a Python driver that verifies the input hash first, runs PDAL, and captures the stage metadata and logs.
  5. Commit the outputs. Push the tiles to the payload store and commit a run record binding input hash, pipeline hash, and output tile hashes.

Working Implementation

The pipeline below reads a LAZ, drops noise and low-confidence returns, reclassifies ground with a progressive morphological filter, and writes a fixed-capacity tile set. Every parameter is explicit so the JSON is a complete, self-describing transform.

{
  "pipeline": [
    {
      "type": "readers.las",
      "filename": "input/acquisition_2026_07_10.laz",
      "tag": "src"
    },
    {
      "type": "filters.range",
      "limits": "Classification![7:7],ReturnNumber[1:5]",
      "tag": "denoise"
    },
    {
      "type": "filters.assign",
      "value": "Classification = 0",
      "tag": "reset_class"
    },
    {
      "type": "filters.smrf",
      "scalar": 1.2,
      "slope": 0.2,
      "threshold": 0.45,
      "window": 16.0,
      "tag": "ground"
    },
    {
      "type": "filters.chipper",
      "capacity": 3000000,
      "tag": "tile"
    },
    {
      "type": "writers.las",
      "filename": "output/tiles/tile_#.laz",
      "compression": "laszip",
      "forward": "all",
      "a_srs": "EPSG:32632"
    }
  ]
}

The Python driver pins the run. It refuses to execute unless the input LAZ matches the recorded hash, hashes the pipeline JSON itself, runs PDAL, and writes a run record that binds inputs to outputs for the commit.

#!/usr/bin/env python3
"""Pinned PDAL driver: verify input hash, run pipeline, record provenance."""
import hashlib
import json
import sys
from pathlib import Path

import pdal


def sha256_file(path: str) -> str:
    """Stream a file through SHA-256 without loading it fully into memory."""
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()


def canonical_pipeline_hash(pipeline_json: str) -> str:
    """Hash the pipeline with keys sorted so formatting never affects the hash."""
    obj = json.loads(pipeline_json)
    canonical = json.dumps(obj, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode()).hexdigest()


def run_pinned(pipeline_path: str, expected_input_hash: str) -> dict:
    pipeline_json = Path(pipeline_path).read_text()
    spec = json.loads(pipeline_json)

    # Locate the reader's input file (first stage with a filename)
    input_path = next(s["filename"] for s in spec["pipeline"]
                      if isinstance(s, dict) and s.get("type", "").startswith("readers"))

    # Gate 1: input must match the pinned hash before we run anything
    actual_input_hash = sha256_file(input_path)
    if actual_input_hash != expected_input_hash:
        sys.exit(f"ABORT: input hash mismatch for {input_path}\n"
                 f"  expected {expected_input_hash}\n  got      {actual_input_hash}")

    # Execute PDAL and capture provenance
    pipeline = pdal.Pipeline(pipeline_json)
    point_count = pipeline.execute()
    metadata = pipeline.metadata          # per-stage CRS, bounds, counts
    log = pipeline.log                    # human-readable run log

    # Hash every output tile for the manifest downstream
    out_dir = Path("output/tiles")
    tile_hashes = {p.name: sha256_file(str(p))
                   for p in sorted(out_dir.glob("tile_*.laz"))}

    run_record = {
        "input_path": input_path,
        "input_hash": actual_input_hash,
        "pipeline_hash": canonical_pipeline_hash(pipeline_json),
        "points_processed": point_count,
        "tile_hashes": tile_hashes,
        "pdal_metadata": json.loads(metadata) if isinstance(metadata, str) else metadata,
    }
    Path("output/run_record.json").write_text(json.dumps(run_record, indent=2))
    return run_record


if __name__ == "__main__":
    # Usage: driver.py pipeline.json <expected_input_sha256>
    record = run_pinned(sys.argv[1], sys.argv[2])
    print(f"Processed {record['points_processed']} points into "
          f"{len(record['tile_hashes'])} tiles")
    print(f"pipeline_hash={record['pipeline_hash'][:12]}…")

Run it from the CLI, passing the input hash you pinned when the acquisition first landed:

# Record the input hash once, at ingest
sha256sum input/acquisition_2026_07_10.laz

# Every subsequent run is gated on that exact hash
python driver.py pipeline.json 4e8f...c17a

# Push tiles to the payload store, commit the pins + record
dvc add output/tiles
git add pipeline.json output/run_record.json output/tiles.dvc
git commit -m "acq 2026-07-10: SMRF ground reclass, 3M-pt tiles"

Validation & Output Verification

Reproducibility is worthless unless you can prove it, so verify the run before committing. First, confirm the pins: the run_record.json must carry both the input hash and the pipeline hash, and re-running canonical_pipeline_hash on the committed JSON must reproduce the recorded pipeline hash. Second, sanity-check the counts — PDAL’s metadata reports points in and points out per stage; assert that the tiled output point total matches the reader count minus whatever filters.range dropped.

# Re-derive the pipeline hash and compare to the committed record
python -c "import json,hashlib; \
o=json.load(open('pipeline.json')); \
c=json.dumps(o,sort_keys=True,separators=(',',':')); \
print(hashlib.sha256(c.encode()).hexdigest())"

# Confirm every tile listed in the run record exists and re-hashes correctly
python -c "import json,hashlib; \
r=json.load(open('output/run_record.json')); \
[print(n, 'OK' if hashlib.sha256(open(f'output/tiles/{n}','rb').read()).hexdigest()==h else 'FAIL') \
for n,h in r['tile_hashes'].items()]"

Finally, do the strongest check available: run the pinned pipeline twice on the same input and confirm the tile hashes are identical between runs. If they diverge, a non-deterministic parameter or PDAL version difference has leaked in, and the run is not yet reproducible.

Failure Modes

  • Symptom: Tile hashes differ between two runs of the same pinned pipeline. Root cause: A filter with order-dependent or threaded non-determinism, or a different PDAL/laz-perf build. Fix: Pin the PDAL version in the environment lock file and confirm filters like filters.smrf produce stable output; hash the sorted tile contents rather than raw bytes if writer ordering varies.
  • Symptom: Driver aborts with an input hash mismatch on a file you believe is unchanged. Root cause: The source LAZ was re-exported or re-flown, changing its bytes even though the survey looks the same. Fix: Treat it as a genuinely new input — record a new input hash and a new commit rather than forcing the old pin.
  • Symptom: Output point count is far below the reader count. Root cause: filters.range limits are too aggressive, silently discarding valid returns. Fix: Inspect PDAL metadata per-stage counts, widen the limits expression, and re-run; never let a filter silently drop the majority of a cloud.
  • Symptom: Tiles reassemble into the wrong CRS or a shifted position. Root cause: Missing a_srs on the writer or forward: "all" omitted, so header CRS/scale/offset were lost. Fix: Set forward: "all" and an explicit a_srs so every tile carries the source spatial reference.

Back to Point-Cloud Versioning and Branching Strategies