Converting Shapefiles to GeoParquet for Diff-Friendly Commits

Turn a legacy multi-file Shapefile into a single byte-reproducible GeoParquet so each commit reflects a real data change — part of GeoParquet vs GeoPackage vs Shapefile for Versioned Workflows.

Concept & Context

A Shapefile fights version control on three fronts at once. It spreads a single layer across four or more sidecar files that must move together; it truncates attribute names to ten characters inside the .dbf; and it leaves the character encoding of text fields undeclared, so the same file decodes differently depending on who reads it. Commit a Shapefile directly and your history fills with churn that no reviewer can interpret.

Converting to GeoParquet solves the structural problems — one columnar file, unbounded field names, embedded CRS — but conversion alone is not enough. A careless writer still emits different bytes on each run because feature order drifts, coordinate precision carries full double noise after a reprojection, or the row-group size changes. The parent guide, GeoParquet vs GeoPackage vs Shapefile for Versioned Workflows, lays out why byte-stability is the property that makes a diff meaningful; this page delivers the concrete pipeline that achieves it for the Shapefile-to-GeoParquet path. Once the output is reproducible, a content-addressable store records exactly one pointer change per real edit, which is precisely what delta tracking algorithms for vector data depend on.

Core Conversion Pipeline

  1. Read with a declared encoding. Look for a .cpg sidecar naming the .dbf encoding; if it is absent, declare one explicitly (commonly CP1252 for legacy files) so accented values decode identically on every machine rather than being guessed per-platform.
  2. Recover truncated field names. The .dbf caps names at ten characters. Apply a truncated-to-full name mapping immediately after load, and assert the rename produces no duplicate target, which would mean two source fields collapsed into one.
  3. Impose canonical ordering. Sort features by a stable primary key when one exists, or by a hash of each geometry’s well-known binary when it does not, so every re-run lays rows down in the same order.
  4. Normalize precision, types, and CRS. Reproject to the target CRS so it embeds as WKT2, snap coordinates to a fixed grid, and coerce each attribute to an explicit dtype so a stray null cannot flip a column between integer and float.
  5. Write with fixed row groups, then verify. Write GeoParquet with a constant row_group_size, then run the whole pipeline twice and compare content hashes before committing the DVC pointer.

Working Implementation

The single script below performs the full conversion. It is deterministic end to end: given the same Shapefile and the same field mapping, it produces byte-identical GeoParquet on every run.

"""Convert a Shapefile to a byte-reproducible GeoParquet for versioned commits."""
import hashlib
import pathlib

import geopandas as gpd
import shapely
from shapely import to_wkb

# --- Configuration (pin every value; changing any of these changes the bytes) ---
SRC_PATH = "input/parcels.shp"
OUT_PATH = "output/parcels.parquet"
DBF_ENCODING = "CP1252"          # override after checking for a .cpg sidecar
TARGET_CRS = "EPSG:4326"
PRECISION_GRID = 0.000001        # ~0.1 m at the equator
ROW_GROUP_SIZE = 100_000         # constant for the life of the dataset
STABLE_KEY = "parcel_id"

# Full names for fields the .dbf truncated to 10 characters.
FIELD_RENAME = {
    "populatio": "population_2020",
    "land_use_c": "land_use_code",
    "last_editd": "last_edited_by",
}


def read_shapefile(path: str, encoding: str) -> gpd.GeoDataFrame:
    """Read a Shapefile with an explicit .dbf encoding (never guess per-platform)."""
    gdf = gpd.read_file(path, encoding=encoding)
    if gdf.crs is None:
        raise ValueError("Source .prj missing or unreadable; set a CRS before converting.")
    return gdf


def recover_field_names(gdf: gpd.GeoDataFrame, rename: dict) -> gpd.GeoDataFrame:
    """Restore full attribute names lost to .dbf 10-char truncation, safely."""
    applicable = {k: v for k, v in rename.items() if k in gdf.columns}
    targets = list(applicable.values())
    if len(targets) != len(set(targets)):
        raise ValueError(f"Rename would create duplicate columns: {targets}")
    gdf = gdf.rename(columns=applicable)
    # Detect any remaining silent truncation collision among original names.
    stems = [c[:10] for c in gdf.columns if c != "geometry"]
    collisions = {s for s in stems if stems.count(s) > 1}
    if collisions:
        raise ValueError(f"Unrecoverable 10-char name collisions: {collisions}")
    return gdf


def canonical_order(gdf: gpd.GeoDataFrame, key: str) -> gpd.GeoDataFrame:
    """Deterministic total ordering so re-runs emit rows in an identical sequence."""
    if key in gdf.columns:
        return gdf.sort_values(key, kind="mergesort").reset_index(drop=True)
    # Fallback: hash the WKB of each geometry (stable, reproducible, key-free).
    digest = gdf.geometry.apply(lambda g: hashlib.sha1(to_wkb(g)).hexdigest())
    return (gdf.assign(_k=digest)
               .sort_values("_k", kind="mergesort")
               .drop(columns="_k")
               .reset_index(drop=True))


def normalize(gdf: gpd.GeoDataFrame) -> gpd.GeoDataFrame:
    """Reproject, snap precision, and pin dtypes so the write is reproducible."""
    gdf = gdf.to_crs(TARGET_CRS)
    gdf["geometry"] = shapely.set_precision(gdf.geometry.values, PRECISION_GRID)
    gdf = gdf.make_valid() if hasattr(gdf, "make_valid") else gdf
    if STABLE_KEY in gdf.columns:
        gdf[STABLE_KEY] = gdf[STABLE_KEY].astype("int64")
    # Coerce object text columns to a stable nullable string dtype.
    for col in gdf.select_dtypes(include="object").columns:
        if col != "geometry":
            gdf[col] = gdf[col].astype("string")
    return gdf


def write_geoparquet(gdf: gpd.GeoDataFrame, path: str) -> str:
    """Write with a fixed row-group size and return the output content hash."""
    pathlib.Path(path).parent.mkdir(parents=True, exist_ok=True)
    gdf.to_parquet(
        path,
        index=False,
        compression="zstd",
        row_group_size=ROW_GROUP_SIZE,
        geometry_encoding="WKB",
    )
    return hashlib.sha256(pathlib.Path(path).read_bytes()).hexdigest()


def convert() -> str:
    gdf = read_shapefile(SRC_PATH, DBF_ENCODING)
    gdf = recover_field_names(gdf, FIELD_RENAME)
    gdf = canonical_order(gdf, STABLE_KEY)
    gdf = normalize(gdf)
    digest = write_geoparquet(gdf, OUT_PATH)
    print(f"Wrote {len(gdf)} features -> {OUT_PATH}\nsha256: {digest}")
    return digest


if __name__ == "__main__":
    convert()

Validation & Output Verification

Byte-reproducibility is a claim you must test, not assume. Run the conversion twice and confirm the two content hashes match:

python convert.py | tee run_a.log
python convert.py | tee run_b.log
diff <(grep sha256 run_a.log) <(grep sha256 run_b.log) && echo "REPRODUCIBLE"

Then confirm no attribute or geometry was lost against the source, and that the CRS embedded correctly:

import geopandas as gpd

src = gpd.read_file("input/parcels.shp", encoding="CP1252")
out = gpd.read_parquet("output/parcels.parquet")

assert len(out) == len(src), f"row count changed: {len(src)} -> {len(out)}"
assert out.crs.to_epsg() == 4326, f"unexpected CRS: {out.crs}"
assert out.geometry.is_valid.all(), "invalid geometry after conversion"
assert "population_2020" in out.columns, "field-name recovery failed"
print("OK:", len(out), "features, EPSG:4326, all valid")

Once both checks pass, commit the pointer rather than the binary:

dvc add output/parcels.parquet
git add output/parcels.parquet.dvc .gitignore
git commit -m "Convert parcels Shapefile to byte-stable GeoParquet"

Failure Modes

  • Symptom: accented attribute values become mojibake (é for é). Root cause: the .dbf was read with the wrong encoding because no .cpg sidecar declared it. Fix: detect the .cpg file or declare encoding= explicitly on read_file, then store output as UTF-8.
  • Symptom: a column vanishes after conversion. Root cause: two source fields truncated to the same ten-character .dbf name and one overwrote the other. Fix: run recover_field_names, which raises on any residual collision so you resolve it at the source before writing.
  • Symptom: the two verification runs produce different hashes. Root cause: feature order was not deterministic, or the row-group size varied. Fix: always call canonical_order before writing and hold ROW_GROUP_SIZE constant.
  • Symptom: DVC re-uploads the whole file after a tiny edit. Root cause: the edit shifted global feature order, rewriting every row group. Fix: keep the canonical sort key stable so an edit touches only the row groups whose features actually changed.

Back to GeoParquet vs GeoPackage vs Shapefile for Versioned Workflows