Partitioning GeoParquet by Region for Smaller Diffs

GeoParquet already keeps an edit from rewriting a whole row group — but the object store versions files, so a one-row change in a national dataset still stores a new copy of the whole thing. This page is a focused companion to GeoParquet vs GeoPackage vs Shapefile for versioned workflows.

Concept & Context

There are two granularities in play and they are easy to conflate. Inside a Parquet file, a column of a row group is the smallest thing a writer rewrites. Outside it, in a content-addressed store, the smallest thing that gets versioned is a file. Columnar layout helps with the first; only partitioning helps with the second.

Hive partitioning — one subdirectory per key value, encoded in the path — turns one logical dataset into many files that a reader still treats as one table. An edit in one region rewrites that region’s file and leaves every other file byte-identical, so the store deduplicates them and the commit costs what the edit costs.

The partition key has to come from how the data is actually edited. A regular grid looks neat and helps nothing when editing follows administrative boundaries, because one county edit lands in several cells. This is the same reasoning that drives splitting a national dataset into stages by region on the compute side, and the two decisions should agree.

Two granularities, and only one of them is versioned Three bands: the row-group band is the smallest thing a columnar writer rewrites, the file band is the smallest thing a content-addressed store versions, and the dataset band is what a reader sees. Partitioning aligns the middle band with the change. SMALLEST WRITE column of a row group SMALLEST VERSIONED UNIT one file — the partition WHAT A READER SEES one logical dataset contained in presented as Columnar layout limits the top band; only partitioning does anything about the middle one.

Core Algorithmic Pipeline

  1. Derive the partition key from the edit pattern in history, not from geography for its own sake.
  2. Sort within each partition by the stable identifier so writes are reproducible.
  3. Pin row-group size, compression and the writer version, since all three affect bytes.
  4. Balance partition sizes into a workable band, splitting large ones by a secondary key.
  5. Verify isolation: a single-feature edit must change exactly one file.

Working Implementation

"""Write a partitioned, byte-stable GeoParquet dataset."""
from __future__ import annotations

from pathlib import Path

import geopandas as gpd
import pyarrow as pa
import pyarrow.parquet as pq

ROW_GROUP_ROWS = 50_000
COMPRESSION = "zstd"
COMPRESSION_LEVEL = 7


def write_partitioned(gdf: gpd.GeoDataFrame, out_dir: str,
                      partition_col: str = "region_id",
                      key: str = "parcel_uid") -> dict[str, int]:
    """One file per partition value, each written deterministically.

    Determinism is the whole point: an unchanged partition must produce
    identical bytes so the content-addressed store recognises it and stores
    nothing new.
    """
    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)

    if gdf[partition_col].isna().any():
        raise ValueError(f"{gdf[partition_col].isna().sum()} row(s) have no "
                         f"{partition_col}; assign every feature before writing")

    counts = {}
    for value, part in gdf.groupby(partition_col, sort=True):
        part = part.sort_values(key, kind="mergesort")        # stable, defined ties

        target = out / f"{partition_col}={value}"
        target.mkdir(parents=True, exist_ok=True)

        table = part.drop(columns=[partition_col]).to_arrow()
        pq.write_table(
            table,
            target / "part-0.parquet",
            row_group_size=ROW_GROUP_ROWS,
            compression=COMPRESSION,
            compression_level=COMPRESSION_LEVEL,
            version="2.6",
            write_statistics=True,
            # Timestamps in the file footer would change on every write and
            # defeat deduplication of an otherwise unchanged partition.
            store_schema=True,
            coerce_timestamps="us",
        )
        counts[str(value)] = len(part)
    return counts


def read_partitioned(path: str, regions: list[str] | None = None) -> gpd.GeoDataFrame:
    """Read the dataset back, pruning partitions when a filter is given."""
    filters = [("region_id", "in", regions)] if regions else None
    return gpd.read_parquet(path, filters=filters)


def rebalance(counts: dict[str, int], low: int = 200_000,
              high: int = 2_000_000) -> list[str]:
    """Partitions outside the workable band, with a suggested action."""
    advice = []
    for value, n in sorted(counts.items()):
        if n > high:
            advice.append(f"{value}: {n:,} rows — split by a secondary key")
        elif n < low:
            advice.append(f"{value}: {n:,} rows — merge with a neighbour")
    return advice

Tracking the dataset as a directory keeps the per-partition granularity all the way through to the object store:

# dvc.yaml
stages:
  publish_partitioned:
    cmd: python scripts/write_partitioned.py --out data/national/parcels.parquet
    deps:
      - data/interim/parcels_reconciled.gpkg
      - scripts/write_partitioned.py
    outs:
      - data/national/parcels.parquet:      # a directory, hashed per file
          cache: true

Because DVC hashes a tracked directory file by file, an unchanged partition contributes an unchanged hash and is not re-uploaded. That is the whole saving, and it depends entirely on the writes being deterministic.

What a one-row edit stores, by layout Horizontal bars comparing bytes stored after editing a single feature under three layouts of the same 40 GB dataset: one file, partitioned by a coarse key, and partitioned by the unit the data is actually maintained in. STORED AFTER A ONE-FEATURE EDIT Single file 40960 MB the store versions files, not row groups Partitioned by grid 1840 MB the edit straddles cells Partitioned by maintenance unit 310 MB The middle bar is why the partition key has to come from the edit pattern rather than from geometry.

Validation & Output Verification

# Two writes of unchanged data must produce identical files everywhere
python scripts/write_partitioned.py --out /tmp/a.parquet
python scripts/write_partitioned.py --out /tmp/b.parquet
diff -r /tmp/a.parquet /tmp/b.parquet && echo "partitioned write is deterministic"

# A single-feature edit must touch exactly one partition file
cp -r /tmp/a.parquet /tmp/edited.parquet
python scripts/edit_one_feature.py --dataset /tmp/edited.parquet \
  --uid EAST-7K3M2P9Q4R --set land_use=industrial
diff -rq /tmp/a.parquet /tmp/edited.parquet | wc -l     # expected: 1
# Partition sizes must stay in the workable band
import json
from partition import rebalance
counts = json.load(open("data/national/parcels.parquet/_counts.json"))
advice = rebalance(counts)
for line in advice:
    print("rebalance:", line)
assert len(advice) < len(counts) * 0.2, "most partitions are outside the size band"

# And the partitioned dataset must read back identically to the source
import geopandas as gpd
from partition import read_partitioned
source = gpd.read_file("data/interim/parcels_reconciled.gpkg").sort_values("parcel_uid")
back = read_partitioned("data/national/parcels.parquet").sort_values("parcel_uid")
assert len(source) == len(back)
assert source["parcel_uid"].tolist() == back["parcel_uid"].tolist()
print("partitioned dataset round-trips")

The one-file diff assertion is the check that matters. If an edit changes two files, either the partition key does not match the edit pattern or the writes are not deterministic — and both make the partitioning cosmetic.

Changing the Partition Key Later

A partition key is not permanent, but changing it is a migration rather than a setting. Every file moves, so the commit that repartitions rewrites the entire dataset once and stores a full second copy in the object store. That is a one-off cost and it is worth paying when the edit pattern has genuinely moved — but it should be a deliberate, separately reviewed commit rather than something that rides along with a data change, or the diff will be unreadable.

The signal that a repartition is due is a rising number of files touched per edit. Track it: if a typical single-region edit began by touching one file and now touches four, either the partitions have drifted out of the size band or responsibility for the ground has been reorganised and the key no longer matches how the data is maintained.

When a partition needs rebalancing A decision ladder over a partition: too large means splitting by a secondary key, too small means merging with a neighbour, and a rising count of files touched per edit means the key itself no longer matches how the data is maintained. Does one partition exceed the size band? Split by a secondary key a local change, not a migration yes no Are several partitions far below it? Merge with neighbours metadata was starting to dominate yes no Has files-touched-per-edit been rising? Repartition on a new key a migration: its own commit, its own review yes no Leave it alone the layout still matches the work Only the third rung rewrites everything, which is why it is worth measuring rather than guessing.

Failure Modes

  • Every partition changes on every write — symptom: no deduplication at all. Root cause: unsorted rows, or writer metadata embedded in the footer. Fix: sort by the stable key and pin the writer settings, as above.

  • One edit rewrites several partitions — symptom: the diff touches four files. Root cause: the partition key does not match how the data is edited. Fix: derive the key from history — which regions changed together in past commits.

  • Thousands of tiny files — symptom: listing the dataset is slower than reading it. Root cause: a partition key with very high cardinality. Fix: partition on a coarser key and use row-group statistics for the finer filtering.

  • A feature has no partition value — symptom: rows silently dropped, or a __HIVE_DEFAULT_PARTITION__ directory appears. Root cause: nulls in the partition column. Fix: fail the write when any value is missing.

Back to GeoParquet vs GeoPackage vs Shapefile for Versioned Workflows