GeoParquet vs GeoPackage vs Shapefile for Versioned Workflows

Choose a vector format that produces meaningful, byte-stable commits instead of noisy binary churn — part of Choosing Formats & Tools for Spatial Data Versioning.

The format you serialize a vector layer into decides how well it lives in a repository. Two of the three formats compared here predate modern version control by decades, and it shows: a single attribute edit can rewrite four files, shuffle feature order, or rewrite a projection string in a way no reviewer asked for. A version-control system cannot tell a genuine change from serialization noise, so every spurious byte becomes a spurious diff, an inflated cache push, and a merge that conflicts for no real reason.

This guide compares GeoParquet, GeoPackage, and Shapefile strictly through a versioning lens — byte-stability of writes, diff-friendliness, single-file versus multi-file layout, columnar versus row storage, coordinate reference system (CRS) storage, attribute typing, on-disk size, and how each behaves under Git and DVC. It then gives a normalization workflow that makes any of the three reproducible enough to commit, with companion deep-dives on converting Shapefiles to GeoParquet for diff-friendly commits and versioning GeoPackage and PostGIS round-trips.

Prerequisites & Environment Setup

Confirm your environment before running any conversion or benchmark below:

• Python 3.10+ with pip or conda
• GeoPandas ≥ 0.14 (its to_parquet/read_parquet path writes GeoParquet 1.0 metadata)
• pyarrow ≥ 14 for explicit row-group control and columnar writes
• GDAL/OGR ≥ 3.6 with the GPKG and Parquet drivers compiled in (ogrinfo --formats | grep -Ei 'parquet|gpkg')
• pyproj ≥ 3.4 so CRS objects serialize as WKT2, not a lossy legacy string
• A fixed precision grid decided up front (for example 0.000001 degrees or 0.001 metres) and a stable primary key on every layer
• Git plus DVC ≥ 3.0 if you intend to track binaries in a content-addressable remote rather than committing them directly

Install the core stack:

pip install "geopandas>=0.14" "pyarrow>=14" "pyproj>=3.4" "shapely>=2.0"

Verify the drivers and versions in one shot:

import geopandas as gpd
import pyarrow as pa
from osgeo import gdal

print("GeoPandas:", gpd.__version__, "| pyarrow:", pa.__version__)
print("Parquet driver:", gdal.GetDriverByName("Parquet") is not None)
print("GPKG driver:", gdal.GetDriverByName("GPKG") is not None)

A CRS decision made once, at ingestion, saves an entire class of diff noise later. Legacy Shapefile projection sidecars store a loose WKT1 string, and different tools re-emit that string with different whitespace and axis-order wording; embedding a canonical WKT2 definition instead removes the ambiguity. The underlying mechanics of what a versioning tool actually records for each change are covered in delta tracking algorithms for vector data.

Core Algorithmic Patterns

Three patterns turn any of these formats from a churn generator into a well-behaved versioned artifact: canonical ordering, deterministic writes, and row-group chunking. The comparison table then maps the raw format properties onto a versioning recommendation.

1. Canonical feature ordering

The single largest source of false diffs is non-deterministic row order. A database export without ORDER BY, a spatial query that returns features in index-traversal order, or a GeoPandas frame rebuilt from a dictionary can all place the same features in a different sequence on each run. Because serialization is order-sensitive, a reordered-but-identical dataset hashes to a completely different value.

Impose a total ordering with O(n log n) cost — sort by a stable primary key when one exists, or by a deterministic spatial key (a geohash or a Hilbert-curve index of the centroid) when it does not. A Hilbert ordering has the pleasant side effect of clustering spatially adjacent features, which improves row-group pruning for GeoParquet readers later.

import geopandas as gpd
from shapely import to_wkb
import hashlib

def canonical_order(gdf, key=None):
    if key and key in gdf.columns:
        return gdf.sort_values(key, kind="mergesort").reset_index(drop=True)
    # Deterministic fallback: hash the well-known-binary of each geometry
    digest = gdf.geometry.apply(lambda g: hashlib.sha1(to_wkb(g)).hexdigest())
    return gdf.assign(_k=digest).sort_values("_k", kind="mergesort") \
              .drop(columns="_k").reset_index(drop=True)

mergesort is specified because it is stable: rows with equal keys keep their prior relative order, so the sort itself never introduces variability.

2. Deterministic writes

Even with fixed ordering, a writer can inject volatility: creation timestamps in file metadata, floating-point coordinates carried at full double precision after a reprojection, or attribute columns whose inferred dtype flips between int64 and float64 when a null appears. Deterministic writing means pinning all of these — round coordinates to the agreed grid, coerce every column to an explicit dtype, and disable any timestamp the driver would otherwise stamp into the container.

3. Row-group chunking

GeoParquet splits a file into row groups, each an independently compressed block with its own column statistics. Chunking is what makes columnar storage version-friendly: an edit confined to one row group leaves the byte ranges of every other row group untouched, so a content-defined store re-transfers only the changed block. Choose a fixed row-group size (50k–150k features is a common sweet spot) and hold it constant across the life of the dataset — changing the size rewrites every group and produces a whole-file diff.

Format comparison

The three formats sit at very different points on the versioning spectrum. The table summarizes the decision, and the diagram after it places each format on axes of structure and diff-stability.

Production Workflow Implementation

The normalization workflow below takes any source layer and produces a byte-stable versioned artifact in five steps. The diagram traces the flow; the numbered steps each carry the code for that stage.

Step 1 — Read and inspect the source

Load the layer once and capture its properties before touching anything. The goal is to know the starting CRS, geometry type, encoding, and field schema so you can assert on them after normalization.

import geopandas as gpd

src = gpd.read_file("source_parcels.shp")
print("CRS:", src.crs)                      # e.g. EPSG:27700
print("geom types:", src.geom_type.unique())
print("dtypes:\n", src.dtypes)
print("rows:", len(src))

Step 2 — Impose canonical ordering

Apply the canonical_order helper from the patterns section. If the source carries a stable identifier, sort by it; otherwise fall back to the geometry hash so the order is reproducible without one.

ordered = canonical_order(src, key="parcel_id")

Step 3 — Fix types, precision, and CRS

Coerce every attribute to an explicit dtype, snap coordinates to the fixed grid, and reproject to the target CRS so the projection is embedded as WKT2 rather than inferred.

import shapely

TARGET_CRS = "EPSG:4326"
GRID = 0.000001   # ~0.1 m at the equator

ordered = ordered.to_crs(TARGET_CRS)
ordered["geometry"] = shapely.set_precision(ordered.geometry.values, GRID)

# Pin attribute dtypes so a stray null cannot flip int64 -> float64
ordered["parcel_id"] = ordered["parcel_id"].astype("int64")
ordered["zone"] = ordered["zone"].astype("string")

Step 4 — Write deterministically

Write to GeoParquet with a fixed row-group size and no lingering index. For teams still on GeoPackage, pin the fid and disable metadata timestamps; the GeoPackage and PostGIS round-trip guide covers that path in full.

ordered.to_parquet(
    "parcels.parquet",
    index=False,
    compression="zstd",
    row_group_size=100_000,     # hold constant for the life of the dataset
    geometry_encoding="WKB",
)

Step 5 — Verify byte-stability and commit

Run the export twice from the same input and confirm the two files hash identically before committing. A content-addressable tool records the pointer only once the bytes settle.

python normalize.py && sha256sum parcels.parquet > a.sum
python normalize.py && sha256sum parcels.parquet > b.sum
diff <(cut -d' ' -f1 a.sum) <(cut -d' ' -f1 b.sum) && echo "byte-stable"

dvc add parcels.parquet
git add parcels.parquet.dvc .gitignore
git commit -m "Add normalized, byte-stable parcels layer"

Code Reliability Patterns

Assert schema and CRS after every conversion

A conversion that silently drops the CRS or reorders columns will pass unit tests but poison downstream joins. Guard the output explicitly:

out = gpd.read_parquet("parcels.parquet")
assert out.crs.to_epsg() == 4326, f"unexpected CRS: {out.crs}"
assert list(out.columns) == ["parcel_id", "zone", "geometry"], "schema drift"
assert out.geometry.is_valid.all(), "invalid geometry after write"

Never edit the versioned file in place

Write to a staging path, validate it, and only then promote it to the tracked location. This preserves the previous good version if any assertion fails and keeps a failed run from corrupting history.

import pathlib, os

staging = pathlib.Path("parcels.staging.parquet")
ordered.to_parquet(staging, index=False, row_group_size=100_000)
if gpd.read_parquet(staging).geometry.is_valid.all():
    os.replace(staging, "parcels.parquet")   # atomic promote
else:
    raise RuntimeError("staging validation failed; tracked file untouched")

Handle Shapefile field-name truncation explicitly

Because the .dbf truncates field names to ten characters, a round-trip through Shapefile can silently collide population_2020 and population_2021 into populatio. When Shapefile is unavoidable as an input, detect the collision before it destroys data:

names = [c[:10] for c in src.columns if c != "geometry"]
dupes = {n for n in names if names.count(n) > 1}
if dupes:
    raise ValueError(f"Shapefile 10-char truncation collides on: {dupes}")

Performance & Scale Considerations

Format choice is also a size and read-speed decision, and both feed directly back into versioning cost — smaller files mean cheaper remote pushes, and columnar reads mean cheaper validation in CI.

The figures below come from a representative 1-million-feature polygon layer (parcel boundaries with twelve attribute columns) on a 2024 workstation (32 GB RAM, NVMe SSD). GeoPackage and GeoParquet were written with the deterministic settings above; the Shapefile is the raw four-file set.

Three effects dominate. First, GeoParquet’s columnar compression roughly triples the storage efficiency of the raw Shapefile, which directly shrinks every DVC push. Second, a validation job that only needs one attribute column reads it from GeoParquet in a fraction of a second because the reader prunes the other columns and skips whole row groups via their statistics. Third, only GeoParquet — and GeoPackage under strict discipline — re-writes to identical bytes, which is the property that makes a diff meaningful at all. For the mechanics of how a versioning system encodes the resulting change, see delta tracking algorithms for vector data.

Troubleshooting & Failure Modes

FAQ

Which format gives the smallest, most reviewable Git diffs?

None of the three produce a text diff you can read line by line — all are binary. GeoParquet is the strongest practical choice: it is a single columnar file whose content hash changes only when the data changes, so a content-addressable tool like DVC records a clean before/after pointer. Shapefile is the worst because one edit touches four or more sidecar files at once, and each sidecar can drift independently.

Why is Shapefile a poor fit for version control?

Shapefile is a multi-file format: .shp, .shx, .dbf, and .prj must stay in lockstep, so a single attribute edit rewrites several files and any missing sidecar corrupts the layer. Field names truncate to ten characters, the .dbf caps text width, and the .prj stores only a loose WKT1 string that many tools rewrite with different whitespace, injecting diffs that represent no real change.

Does GeoPackage produce stable diffs across exports?

Only if you control the write. GeoPackage is a SQLite database, so page layout, the auto-incrementing fid, and internal metadata timestamps can shift between dumps even when the data is identical. Fix the feature ordering, pin the fid column, and strip volatile metadata to make successive exports byte-comparable — the GeoPackage and PostGIS round-trip guide walks through exactly which knobs to set.

How does columnar storage help versioned analytics?

GeoParquet stores each attribute column contiguously and splits rows into independent row groups. A change confined to one region or one column rewrites only the affected row groups, so a content-aware store transfers less data on each commit. Columnar layout also lets a validation job pull a single field without decoding whole features, which makes CI checks dramatically cheaper.

Can I keep Shapefile as a delivery format but version something else?

Yes, and it is the recommended pattern. Version a normalized GeoParquet or GeoPackage as the source of truth, then generate Shapefile on demand for consumers that still require it. The generated Shapefile stays a disposable build artifact, never committed, so its multi-file instability never pollutes your history.

Related

• Converting Shapefiles to GeoParquet for Diff-Friendly Commits — a full pipeline that fixes encoding, field truncation, and ordering before the write
• Versioning GeoPackage and PostGIS Round-Trips — keeping a GeoPackage dump byte-stable across repeated exports from PostGIS
• Delta Tracking Algorithms for Vector Data — how a versioning system encodes the change between two format snapshots
• Large File Handling in DVC for GIS — remote configuration and cache strategy for the binary artifacts these formats produce

Back to Choosing Formats & Tools for Spatial Data Versioning