Hashing Spatial Datasets for Reproducible Fingerprints

A byte hash tells you whether two files are identical; a spatial fingerprint tells you whether two datasets say the same thing — and in a versioned pipeline the second question is almost always the one being asked. This page is a focused companion to provenance and lineage tracking for spatial pipelines.

Concept & Context

Every content-addressed system hashes bytes, and for most artifacts that is the right thing to do. Spatial datasets break the assumption because the same features can be written to disk in many byte sequences that are all correct. Feature order follows the writer’s iteration. Coordinate precision follows the driver’s default. Compression level, block size and even an embedded creation timestamp all vary without any feature changing.

The consequence is familiar to anyone who has watched a repository grow while the data stood still: a nightly export produces a new hash every night, the storage layer dutifully keeps every version, and nobody can tell from the hashes which nights actually changed something. The format normalisation work that makes writes deterministic solves much of this at the file level; a content fingerprint solves the rest, and keeps working across format conversions where byte determinism cannot.

A content fingerprint is computed over the features, not the container. Two datasets with the same features in a different order, or written by different drivers, produce the same fingerprint. That is precisely what makes it usable as a lineage identifier and as the input to a “did anything actually change” check.

What each kind of hash actually answers A grid comparing a byte hash with a content fingerprint across four operations — re-encoding, reordering features, converting format, and editing one attribute — showing which operations change each hash. Byte hash Content fingerprint Re-encode, same data changes unchanged Reorder features changes unchanged Convert to another format changes unchanged Edit one attribute changes changes, and names the feature Both are worth keeping: one answers whether these bytes are cached, the other whether this is the same data.

Core Algorithmic Pipeline

  1. Sort by a stable key. Feature order is a storage artefact. Sort by the identifier the layer guarantees, and refuse to fingerprint a layer that has no such identifier rather than falling back to row order.
  2. Quantise coordinates to the declared precision of the data. Everything below that is arithmetic noise that changes with every reprojection.
  3. Normalise the geometry encoding. Well-known binary with a fixed byte order and a fixed dimensionality, so the same shape produces the same bytes regardless of the library that wrote it.
  4. Serialise attributes canonically — sorted keys, explicit null marker, a fixed numeric format — so a JSON writer’s key ordering cannot change the result.
  5. Digest per feature, then fold. Hash each feature independently, then hash the ordered sequence of digests. The intermediate digests are what let you say which feature changed rather than only that something did.
From features to one comparable value A chain of five stages: sort by the stable key, quantise coordinates to the declared precision, normalise the geometry encoding, digest each feature independently, then fold the ordered digests into one layer fingerprint. Sort by the stable key ordered Quantise to survey precision snapped Normalise fixed WKB encoding bytes Digest one per feature digests Fold one per layer The per-feature digests survive the fold, which is what lets a comparison name the feature that changed.

Working Implementation

"""Content fingerprint for a vector layer: stable across drivers, compression
and feature order, sensitive to any actual change in geometry or attributes."""
from __future__ import annotations

import hashlib
import json

import geopandas as gpd
from shapely import set_precision
from shapely.wkb import dumps as wkb_dumps

NULL = "\x00NULL\x00"


def canonical_attributes(row, skip=("geometry",)) -> str:
    """Sorted keys, explicit nulls, fixed numeric formatting."""
    items = []
    for key in sorted(k for k in row.index if k not in skip):
        value = row[key]
        if value is None or (isinstance(value, float) and value != value):
            items.append((key, NULL))
        elif isinstance(value, float):
            items.append((key, f"{value:.10g}"))       # no repr drift across versions
        else:
            items.append((key, str(value)))
    return json.dumps(items, ensure_ascii=False, separators=(",", ":"))


def feature_digest(row, grid: float) -> str:
    """One digest per feature, over quantised geometry plus canonical attributes."""
    geom = row.geometry
    if geom is None or geom.is_empty:
        geom_bytes = b"\x00EMPTY"
    else:
        # Quantise to the precision the data actually has, and pin the encoding:
        # little-endian WKB, 2D, no SRID embedded (the CRS is hashed separately).
        snapped = set_precision(geom, grid)
        geom_bytes = wkb_dumps(snapped, hex=False, output_dimension=2,
                               byte_order=1, include_srid=False)

    h = hashlib.blake2b(digest_size=16)
    h.update(geom_bytes)
    h.update(b"\x1f")                                   # unit separator
    h.update(canonical_attributes(row).encode("utf-8"))
    return h.hexdigest()


def layer_fingerprint(path: str, key: str, grid: float = 0.001, layer: str | None = None):
    """Return (fingerprint, per-feature digests keyed by identifier).

    grid is the quantisation step in CRS units — 0.001 for a metric CRS means
    millimetre precision, which is finer than any field survey and coarser than
    floating-point noise.
    """
    gdf = gpd.read_file(path, layer=layer)

    if key not in gdf.columns:
        raise ValueError(
            f"{path} has no stable key column {key!r}; fingerprinting by row "
            "order would produce a value that changes on every re-export"
        )
    if gdf[key].isna().any() or gdf[key].duplicated().any():
        raise ValueError(f"{key!r} must be non-null and unique to fingerprint {path}")

    gdf = gdf.sort_values(key, kind="mergesort")        # stable sort, defined ties

    digests = {
        str(row[key]): feature_digest(row, grid)
        for _, row in gdf.iterrows()
    }

    roll = hashlib.blake2b(digest_size=32)
    roll.update(f"gdv-fingerprint/1|crs={gdf.crs.to_string()}|grid={grid:g}"
                .encode("utf-8"))
    for fid in sorted(digests):
        roll.update(fid.encode("utf-8"))
        roll.update(b"\x1e")                            # record separator
        roll.update(digests[fid].encode("ascii"))
    return roll.hexdigest(), digests


def changed_features(before: dict, after: dict):
    """Which features actually differ, from two per-feature digest maps."""
    added = sorted(set(after) - set(before))
    removed = sorted(set(before) - set(after))
    modified = sorted(k for k in set(before) & set(after) if before[k] != after[k])
    return {"added": added, "removed": removed, "modified": modified}

The CRS goes into the rolling hash rather than into each feature. Two layers holding identical coordinates in different reference systems are not the same dataset, and folding the CRS in once makes that explicit without paying for it per feature.

Validation & Output Verification

The properties worth asserting are the ones the fingerprint exists to provide:

import subprocess
from fingerprint import layer_fingerprint, changed_features

fp_gpkg, digests = layer_fingerprint("data/parcels.gpkg", key="parcel_id")

# 1. Stable across a format conversion — same features, different container
subprocess.run(["ogr2ogr", "-f", "Parquet", "/tmp/parcels.parquet",
                "data/parcels.gpkg"], check=True)
fp_parquet, _ = layer_fingerprint("/tmp/parcels.parquet", key="parcel_id")
assert fp_gpkg == fp_parquet, "fingerprint must survive a format change"

# 2. Stable across feature reordering
subprocess.run(["ogr2ogr", "-f", "GPKG", "/tmp/shuffled.gpkg", "data/parcels.gpkg",
                "-sql", "SELECT * FROM parcels ORDER BY random()"], check=True)
fp_shuffled, _ = layer_fingerprint("/tmp/shuffled.gpkg", key="parcel_id")
assert fp_gpkg == fp_shuffled, "fingerprint must not depend on feature order"

# 3. Sensitive to a real change, and able to name it
subprocess.run(["ogr2ogr", "-f", "GPKG", "/tmp/edited.gpkg", "data/parcels.gpkg",
                "-sql", "SELECT * FROM parcels"], check=True)
subprocess.run(["ogrinfo", "/tmp/edited.gpkg", "-sql",
                "UPDATE parcels SET land_use='industrial' "
                "WHERE parcel_id='EAST-7K3M2P9Q4R'"], check=True)
fp_edited, edited_digests = layer_fingerprint("/tmp/edited.gpkg", key="parcel_id")
assert fp_edited != fp_gpkg
assert changed_features(digests, edited_digests)["modified"] == ["EAST-7K3M2P9Q4R"]
print("fingerprint stable under re-encoding, sensitive to content")

These three assertions belong in the repository’s test suite, not in a notebook. They are the definition of what the fingerprint promises, and they fail loudly the day someone changes the quantisation grid or the serialisation.

The three assertions that define the fingerprint Two panels. The first lists what the fingerprint must be blind to: encoding, feature order and container format. The second lists what it must detect: an attribute edit, a geometry move beyond the grid, and an added or removed feature. MUST IGNORE Compression level and driver version The order features happen to be written in The container format the data sits in Coordinate digits below the survey precision MUST DETECT Any attribute value that changed Any geometry move larger than the grid Any feature added or removed A change of coordinate reference system Both columns belong in the test suite — the second one fails when the grid is set too coarse. A fingerprint tested only on the left column will happily hide real edits.

Failure Modes

  • The fingerprint changes on every export — symptom: nightly runs produce a new value with no edits. Root cause: sorting omitted, or coordinates hashed at full floating-point precision. Fix: sort by the stable key and quantise before hashing; verify with the reordering assertion above.

  • Two genuinely different datasets share a fingerprint — symptom: a change is not detected. Root cause: the quantisation grid is coarser than the edits being made. Fix: set the grid from the survey precision, not from convenience; a 1 m grid on a layer edited at centimetre scale hides real work.

  • Fingerprinting fails on a legacy layer — symptom: the key-column check raises. Root cause: no stable identifier exists. Fix: mint one and store it, as covered in designing conflict-free identifiers for offline capture; do not fall back to row order.

  • Fingerprints disagree between two machines — symptom: CI computes a different value from a workstation. Root cause: a library version whose WKB output or float formatting differs. Fix: pin the geometry library, record its version alongside the fingerprint, and treat a version bump as a fingerprint migration.

Back to Provenance and Lineage Tracking for Spatial Pipelines