Splitting a National Dataset into DVC Stages by Region

When one county corrects a boundary and the pipeline re-processes the whole country, the problem is not compute β€” it is that the pipeline’s unit of work does not match the data’s unit of change. This page is a focused companion to large file handling in DVC for GIS.

Concept & Context

A pipeline stage re-runs when any of its declared dependencies change. A stage that takes a national layer as its dependency therefore re-runs on any edit anywhere, and a stage that takes four hours turns a five-minute correction into a four-hour wait β€” which teaches people to batch corrections, which makes reviews larger and merges harder.

Partitioning aligns the stage boundary with the change boundary. If parcels are maintained by county and edits arrive per county, then a per-county stage re-runs only for the county that changed. Everything else reports up to date, and the national product is rebuilt by a cheap join.

The partition has to come from the edit pattern rather than from geography for its own sake. Partitioning by a regular grid looks tidy and helps nothing if editing follows administrative boundaries, because a single county edit will touch several grid cells. The right question is not β€œhow do I divide this map” but β€œwhat unit does this data change in”.

What re-runs after a one-county edit Two rows of stage cells. The top row is a single national stage, where one county's edit re-runs the whole thing. The bottom row is per-region stages, where the same edit re-runs one stage and the join, and every other region reports up to date. r1 r2 r3 r4 r5 r6 r7 join one stage per region Ξ” re-runs, whether or not it needed to the region that actually changed the join, which always re-runs reports up to date The join is the only unavoidable cost, which is the argument for keeping it to concatenation.

Core Algorithmic Pipeline

  1. Establish the edit pattern from history: which regions changed in each of the last few dozen commits.
  2. Split the source once into per-region inputs by a deterministic assignment rule.
  3. Express the per-region work as a foreach stage, so one definition covers every region.
  4. Join in a separate stage that concatenates rather than recomputes.
  5. Verify equivalence against a whole-layer run, especially at region boundaries.

Working Implementation

# dvc.yaml β€” one definition per unit of work, expanded per region
stages:
  partition:
    cmd: python scripts/partition.py --input data/raw/parcels.gpkg --out data/by_region
    deps:
      - data/raw/parcels.gpkg
      - scripts/partition.py
      - config/regions.json
    outs:
      - data/by_region

  process:
    foreach: ${regions}
    do:
      cmd: >-
        python scripts/process_region.py
          --input data/by_region/${item}.gpkg
          --output data/processed/${item}.gpkg
          --buffer-context data/by_region
          --tolerance ${quantisation.grid_m}
      deps:
        - data/by_region/${item}.gpkg
        - scripts/process_region.py
      params:
        - quantisation.grid_m
      outs:
        - data/processed/${item}.gpkg

  join:
    cmd: python scripts/join_regions.py --in data/processed --out data/national/parcels.gpkg
    deps:
      - data/processed
      - scripts/join_regions.py
    outs:
      - data/national/parcels.gpkg
# scripts/partition.py
"""Assign every feature to exactly one region, deterministically."""
from __future__ import annotations

import json
from pathlib import Path

import geopandas as gpd


def partition(input_path: str, regions_path: str, out_dir: str) -> dict[str, int]:
    gdf = gpd.read_file(input_path)
    regions = gpd.read_file(regions_path).to_crs(gdf.crs)

    # Representative point, not centroid: a centroid can fall outside a concave
    # polygon and be assigned to the wrong region β€” or to none at all.
    anchors = gdf.copy()
    anchors["geometry"] = gdf.geometry.representative_point()

    assigned = gpd.sjoin(anchors, regions[["region_id", "geometry"]],
                         how="left", predicate="within")

    unassigned = assigned["region_id"].isna().sum()
    if unassigned:
        raise ValueError(
            f"{unassigned} feature(s) fall outside every region; the region "
            "layer does not cover the data extent"
        )

    gdf["region_id"] = assigned["region_id"].to_numpy()

    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)
    counts = {}
    for region_id, part in gdf.groupby("region_id"):
        # Sort within the partition so a re-partition of unchanged data
        # produces byte-identical files and the stage reports up to date.
        part = part.sort_values("parcel_uid", kind="mergesort")
        part.to_file(out / f"{region_id}.gpkg", driver="GPKG")
        counts[str(region_id)] = len(part)

    total = sum(counts.values())
    if total != len(gdf):
        raise ValueError(f"partition lost features: {len(gdf)} in, {total} out")

    (out / "manifest.json").write_text(json.dumps(counts, indent=2, sort_keys=True))
    return counts

The regions list lives in params.yaml so the foreach expansion is itself versioned:

# params.yaml
regions: [north, north-east, midlands, east, south-east, south-west, wales, scotland]
quantisation:
  grid_m: 0.001

Adding a region is then a reviewed parameter change that adds one stage, rather than an edit to a generated pipeline file.

The context a per-region job still needs Two panes over the same regional boundary. Without context, a smoothing operation sees only its own region and produces a discontinuity at the edge. With a buffer of neighbouring features passed in and the result clipped back, the boundary matches what a whole-layer run would produce. NO CONTEXT discontinuity The feature continues past the edge the job could not see. BUFFERED CONTEXT Read the buffer, process, then clip the output back to the region. The buffer is read but never written, so the union of outputs is still exactly the input set.

Validation & Output Verification

The partition is only safe if it is lossless and boundary-neutral:

# Every feature lands in exactly one partition
python - <<'PY'
import json, geopandas as gpd
counts = json.load(open("data/by_region/manifest.json"))
whole = len(gpd.read_file("data/raw/parcels.gpkg"))
assert sum(counts.values()) == whole, f"{whole} in, {sum(counts.values())} out"
print(f"partition is lossless across {len(counts)} regions")
PY

# The joined national product must match a whole-layer run
python scripts/process_whole.py --input data/raw/parcels.gpkg --output /tmp/whole.gpkg
python - <<'PY'
from fingerprint import layer_fingerprint
a, _ = layer_fingerprint("data/national/parcels.gpkg", key="parcel_uid")
b, _ = layer_fingerprint("/tmp/whole.gpkg", key="parcel_uid")
assert a == b, "partitioned result differs from the whole-layer result"
print("partitioned and whole-layer runs agree")
PY

# One region's edit must re-run one region's stage
touch data/by_region/midlands.gpkg
dvc status | grep -c "changed deps"
dvc repro --dry 2>&1 | grep -E "^Running stage" | sed 's/.*process@//'
# expected: midlands only

The equivalence check belongs in CI on a small fixture. Partitioned pipelines drift towards subtle boundary differences β€” a smoothing operation that had national context and now has regional context β€” and the fingerprint comparison is what catches it.

Three properties a partitioned pipeline has to keep proving Three checks: the partition is lossless, the joined result is identical to a whole-layer run, and a single region's edit re-runs only that region's stage. 1 Lossless partition features in equals features out 2 Equivalent to a whole-layer run fingerprints match, boundaries included 3 Isolated re-runs one edit, one region's stage The middle check is the one that catches boundary drift, and it belongs in CI on a small fixture.

What the Join Stage Should and Should Not Do

The join is the one stage that depends on every region, so it re-runs whenever any region does β€” which makes it the natural place for work to accumulate and the worst place for it. Keep it to concatenation, a stable sort and a single write. Anything that recomputes across the whole dataset there undoes the partitioning entirely, because that computation now runs on every edit anywhere in the country.

Where a genuinely national computation is required β€” a nationwide topology check, or a summary that cannot be derived per region β€” give it its own stage downstream of the join, and let it depend on the joined output rather than on the regions. It still re-runs on any change, but it is then one clearly labelled expensive stage rather than a cost hidden inside the assembly step, and the pipeline’s status output shows honestly where the time is going.

Failure Modes

  • Results differ near region boundaries β€” symptom: geometry along a county line changes after partitioning. Root cause: an operation needing neighbouring context now sees only one region. Fix: pass a context buffer to each job, as --buffer-context does, and clip the output back to the region.

  • Every region re-runs on any edit β€” symptom: partitioning bought nothing. Root cause: per-region stages depend on the whole partition directory rather than on their own file. Fix: declare data/by_region/${item}.gpkg as the dependency, not the directory.

  • The partition stage never reports up to date β€” symptom: a full re-partition on every run. Root cause: non-deterministic writes, so the partition outputs get new hashes each time. Fix: sort within each partition before writing, as above.

  • Features vanish β€” symptom: the national product has fewer features than the source. Root cause: a region layer that does not fully cover the data extent, so some features are assigned to nothing. Fix: fail the partition when any feature is unassigned rather than dropping it.

Back to Large File Handling in DVC for GIS