CI/CD Validation Pipelines for Spatial Repositories

A validation pipeline is the automated gate that refuses to let a broken geometry, an undeclared coordinate reference system, or a 4 GB accidental commit reach your main branch. This guide is part of Geospatial Data Versioning Fundamentals & Architecture.

Spatial repositories fail in ways that generic software CI never anticipates. A contributor reprojects a layer, forgets to write the CRS back, and every downstream join silently offsets by tens of metres. Another commits a self-intersecting polygon that passes visual inspection but crashes the nightly tiling job. A third drags a raw LiDAR tile into the repo and blows past the hosting quota. None of these produce a compile error, a failing unit test, or a red line in a text diff. The only reliable defense is a pipeline that opens each changed layer, inspects its geometry and metadata, and blocks the merge when something is wrong.

This page shows how to assemble that pipeline from two enforcement points — pre-commit hooks on the developer’s machine and CI gates on the server — driven by a single reusable validation script. We wire the script into both GitHub Actions and GitLab CI, run it against DVC-managed artifacts, and order the checks so contributors get feedback in seconds. Two companion deep-dives extend the pattern: a topology validation gate in GitHub Actions that annotates the pull request inline, and a GitLab pipeline for automating DVC push on merge that promotes and tags the validated data version.

Spatial CI/CD Validation Pipeline Stages A left-to-right pipeline: checkout the pull request, restore the DVC cache and pull artifacts, run the validation script across changed layers, evaluate a fail-fast gate, then either merge on pass or block for review on failure. Checkout PR / MR dvc pull restore cache Validate changed layers Fail-fast gate Merge pass Block review ✓ ✗ Geometry validity, CRS, schema, topology, and file-size checks all run inside the Validate stage; any critical finding trips the gate.

Prerequisites & Environment Setup

The pipeline runs the same validation code in three places — a developer’s shell, a pre-commit hook, and a CI runner — so the environment must be reproducible everywhere. Establish this baseline before writing any workflow YAML:

Install the toolchain and register the hook:

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt        # pinned versions only
pip install pre-commit
pre-commit install                     # registers the git hook
dvc remote list                        # confirm the remote is reachable

Core Pipeline Patterns

Three design decisions separate a pipeline that scales from one that becomes a bottleneck teams learn to bypass.

1. Fail-fast gate ordering

Order checks from cheapest to most expensive and stop at the first critical failure. A missing CRS is detectable the moment the file header is read; a file-size violation is a single os.stat call; a topology overlay on tens of thousands of polygons can take a minute. Because the overwhelming majority of bad commits trip a cheap check, running those first means most failures surface in well under a second. Only commits that clear every cheap gate pay for the expensive topology pass. Model complexity roughly as O(1) header reads, then O(n) per-feature validity, then O(n log n) indexed topology overlay — and run them strictly in that order.

2. Matrix over layers

A multi-layer GeoPackage or a directory of layers should fan out into an independent job per layer rather than a single serial loop. In CI this is a build matrix: enumerate the changed layers, emit them as a JSON array, and let the runner schedule one job per entry. Failures are isolated — a broken parcels layer does not mask a valid roads layer — and the jobs run in parallel across the runner pool, so total wall-clock time is bounded by the slowest single layer rather than the sum of all layers.

3. Caching the DVC remote

Restoring the DVC cache between runs is the single largest speedup available. Key the cache on the content hash recorded in the .dvc files so an unchanged artifact is a cache hit and never re-downloaded. Combined with fetching only changed layers, a repository whose full data footprint is several gigabytes typically transfers only the handful of megabytes that actually changed in the pull request. The table below summarizes the three patterns and their impact.

Pattern Mechanism Primary benefit
Fail-fast ordering Cheap checks first, stop on critical Median failure surfaces in < 1 s
Layer matrix One CI job per changed layer Parallelism + isolated failures
DVC remote cache Cache keyed on content hash Skips re-download of unchanged bytes

Production Workflow Implementation

Step 1: Author the reusable validation script

The heart of the pipeline is one Python entry point that every enforcement layer calls. It opens each layer, runs the ordered checks, and exits non-zero on any critical finding so a CI step naturally fails.

#!/usr/bin/env python3
"""validate_spatial.py — single source of truth for spatial repo validation."""
import sys
import os
import json
import yaml
import geopandas as gpd
import fiona
from pyproj import CRS
from shapely.validation import explain_validity


def load_config(path: str = "validation.yml") -> dict:
    with open(path) as fh:
        return yaml.safe_load(fh)


def check_file_size(path: str, max_mb: float) -> list[dict]:
    size_mb = os.path.getsize(path) / (1024 * 1024)
    if size_mb > max_mb:
        return [{"check": "file_size", "severity": "critical",
                 "message": f"{path} is {size_mb:.1f} MB (limit {max_mb} MB)"}]
    return []


def check_layer(gdf: gpd.GeoDataFrame, layer: str, cfg: dict) -> list[dict]:
    findings: list[dict] = []

    # --- CRS consistency (cheap) ---
    if gdf.crs is None:
        findings.append({"check": "crs", "severity": "critical", "layer": layer,
                         "message": "layer has no CRS defined"})
    else:
        want = CRS.from_epsg(cfg["target_epsg"])
        if not CRS.from_user_input(gdf.crs).equals(want):
            findings.append({"check": "crs", "severity": "critical", "layer": layer,
                             "message": f"CRS {gdf.crs} != EPSG:{cfg['target_epsg']}"})

    # --- Attribute schema (cheap) ---
    required = set(cfg.get("required_fields", {}).get(layer, []))
    missing = required - set(gdf.columns)
    if missing:
        findings.append({"check": "schema", "severity": "critical", "layer": layer,
                         "message": f"missing required fields: {sorted(missing)}"})

    # --- Geometry validity + topology (expensive, runs last) ---
    invalid = gdf[~gdf.geometry.is_valid]
    for idx, geom in invalid.geometry.items():
        findings.append({"check": "topology", "severity": "critical", "layer": layer,
                         "message": f"feature {idx}: {explain_validity(geom)}"})
    return findings


def validate(path: str, cfg: dict) -> list[dict]:
    findings = check_file_size(path, cfg["max_file_mb"])
    if any(f["severity"] == "critical" for f in findings):
        return findings                      # fail fast before opening the file
    for layer in fiona.listlayers(path):
        gdf = gpd.read_file(path, layer=layer)
        findings.extend(check_layer(gdf, layer, cfg))
    return findings


def main(paths: list[str]) -> int:
    cfg = load_config()
    all_findings: list[dict] = []
    for path in paths:
        all_findings.extend(validate(path, cfg))
    print(json.dumps(all_findings, indent=2))
    blocking = [f for f in all_findings if f["severity"] == "critical"]
    return 1 if blocking else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

Step 2: Wire the fast subset into pre-commit

The hook runs only the cheap checks locally so a push never carries an obvious defect. Point it at the same script with a --fast convention (or a separate config that skips the topology pass) to keep the developer feedback loop instant.

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: spatial-validate
        name: spatial layer validation (fast)
        entry: python validate_spatial.py
        language: system
        files: '\.(gpkg|geojson)$'
        pass_filenames: true

Step 3: Gate on every pull request in GitHub Actions

The CI workflow checks out the branch, restores the DVC cache, pulls the real artifacts, computes the changed layers, and runs the full validator. Marking this check run required in branch protection is what actually blocks the merge.

name: Spatial Validation
on:
  pull_request:
    branches: [main]
    paths: ["**/*.gpkg", "**/*.geojson", "validation.yml"]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: "pip"

      - name: Install pinned toolchain
        run: pip install -r requirements.txt

      - name: Cache DVC remote
        uses: actions/cache@v4
        with:
          path: .dvc/cache
          key: dvc-${{ hashFiles('**/*.dvc') }}
          restore-keys: dvc-

      - name: Pull spatial artifacts
        run: dvc pull --quiet
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.DVC_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.DVC_SECRET }}

      - name: Collect changed layers
        id: changed
        run: |
          git diff --name-only origin/${{ github.base_ref }} HEAD \
            | grep -E '\.(gpkg|geojson)$' > changed.txt || true
          echo "files=$(tr '\n' ' ' < changed.txt)" >> "$GITHUB_OUTPUT"

      - name: Validate (fail-fast gate)
        if: steps.changed.outputs.files != ''
        run: python validate_spatial.py ${{ steps.changed.outputs.files }}

Step 4: Mirror the gate in GitLab CI

Teams on GitLab wire the identical script into a .gitlab-ci.yml merge-request pipeline. The cache stanza restores .dvc/cache across runs and the rules clause scopes the job to merge requests only.

stages: [validate]

spatial-validate:
  stage: validate
  image: python:3.11-slim
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  cache:
    key: "dvc-$CI_COMMIT_REF_SLUG"
    paths: [.dvc/cache]
  variables:
    AWS_ACCESS_KEY_ID: $DVC_KEY_ID
    AWS_SECRET_ACCESS_KEY: $DVC_SECRET
  before_script:
    - pip install -r requirements.txt
    - dvc pull --quiet
  script:
    - git fetch origin $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
    - CHANGED=$(git diff --name-only origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME HEAD | grep -E '\.(gpkg|geojson)$' || true)
    - if [ -n "$CHANGED" ]; then python validate_spatial.py $CHANGED; fi

The same discipline that powers spatial conflict interception — see automated conflict detection in merge requests — applies here: the gate must be a required status check, never an advisory one.

Code Reliability Patterns

A validation gate that flaps between pass and fail on identical input erodes trust faster than no gate at all. Build the checks to be idempotent and deterministic.

Pin the numeric toolchain, not just the API. geopandas>=0.14 is not enough. GEOS 3.11 and 3.12 return subtly different make_valid output for the same self-intersection. Pin exact versions in requirements.txt and rebuild the runner image only on a deliberate upgrade.

Snap before you compare. Any check that compares geometry across two revisions must round coordinates to a fixed precision grid — 6 decimals for EPSG:4326, 3 for a metre-based projected CRS — so floating-point residue cannot flip a result. Snapping makes the topology check idempotent: running it twice on the same input yields the same finding set.

Never rely on implicit reprojection. Assert the CRS equals the configured EPSG target and fail if it does not. Silently calling to_crs inside a check hides the very drift the check exists to catch.

Make the exit code the contract. The script’s only side effect that CI depends on is its exit status. Print findings as JSON to stdout for humans, but let the return code — non-zero on any critical finding — be the single, deterministic signal the gate reads.

Guard against empty pulls. If dvc pull restores a placeholder instead of real bytes, gpd.read_file may open a zero-feature layer that passes every check vacuously. Assert a non-zero feature count as part of validation so a broken artifact restore fails loudly rather than passing silently.

Performance & Scale Considerations

Validate only changed files. The git diff --name-only step against the merge base is the highest-leverage optimization. A repository with 400 layers where a pull request touches two should validate two, not 400. Everything else in the pipeline scales from this.

Cache the DVC remote aggressively. Keying the cache on hashFiles('**/*.dvc') means unchanged artifacts are restored from local cache in seconds instead of pulled over the network. On a repository with a 6 GB data footprint, a typical pull request transfers only the few megabytes that changed.

Fan out with a layer matrix. For a single large multi-layer package, emit the changed layers as a matrix and let jobs run in parallel. Wall-clock time drops to the slowest individual layer rather than the serial sum.

Bound memory per job. Overlay-heavy topology checks scale with feature count. Tile layers above 500 MB by bounding-box grid and validate each tile in a separate job to keep peak memory under 1 GB.

Benchmark reference. On a 4-core runner with 8 GB RAM, the full validator processes a 45 MB GeoPackage of roughly 70,000 polygons in about 55 seconds. Restricting to changed files plus a warm DVC cache brings a typical single-layer pull request in under 20 seconds end to end.

Troubleshooting & Failure Modes

Symptom Root cause Fix
Validation passes locally but fails in CI Unpinned GEOS/PROJ differ between machines Pin exact shapely, pyproj, fiona versions and rebuild the runner image
Checks run against empty layers, always pass dvc pull restored pointer files, not bytes Add DVC remote credentials to CI secrets; assert a non-zero feature count
Pipeline validates hundreds of unchanged layers No changed-files filter on the run Diff against origin/<base> and pass only matching paths to the validator
Topology check times out on large layers Whole-layer overlay without indexing or tiling Tile by bounding-box grid; run one CI job per tile in parallel
Gate reports failure but the merge proceeds Check run not marked required in branch protection Add the job to required status checks so a red run blocks merge
Intermittent sliver topology failures Floating-point residue after reprojection Snap coordinates to a fixed precision grid before the validity check

FAQ

Should validation live in pre-commit hooks or in CI? Both, running the same script. Pre-commit runs the fast checks — geometry validity, CRS presence, file size — locally so contributors catch defects in seconds. CI re-runs the full suite, including expensive topology overlays, on a clean runner where nobody can skip the hook. Treat CI as the authoritative gate and pre-commit as a courtesy that shortens the feedback loop.

How do I keep the pipeline from re-validating unchanged layers? Diff the changed paths from the merge base with git diff --name-only and feed only matching spatial files into the validator. Combine that with a cached DVC remote so unchanged binary artifacts are restored from cache rather than re-downloaded. On large monorepos this turns a ten-minute run into a sub-minute one.

Why order the validation checks fail-fast? Cheap checks catch the most common mistakes. A missing CRS or an oversized file is detectable in milliseconds, while a topology overlay on 80,000 polygons can take a minute. Running the cheap checks first means the majority of bad commits fail almost instantly, saving runner minutes and giving contributors near-immediate feedback.

How do I make the geometry checks deterministic across runners? Pin every spatial library to an exact version, snap coordinates to a fixed precision grid before comparison, and never rely on implicit CRS transformation. GEOS and PROJ change numerical output between releases, so an unpinned runner can pass a commit that a colleague’s machine rejects. Pin the toolchain and the checks become reproducible.

Where do DVC or Git-LFS artifacts fit into the CI run? The repository stores lightweight pointer files; the actual GeoPackage or raster bytes live on a remote. CI must restore the DVC cache and run dvc pull before validation, otherwise the checks run against empty placeholders. After a merge to the main branch, a separate job runs dvc push so the canonical artifact is preserved on the shared remote.