Schema Drift Reconciliation for Spatial Layers

Reconciling schema drift means bringing the attribute and geometry definitions of a vector layer back into agreement after they have diverged across branches or time — part of the broader Conflict Resolution & Team Synchronization Workflows that keep collaborative spatial repositories mergeable.

Row-level conflicts get most of the attention, but the more corrosive problem is drift in the shape of the table. One branch adds a risk_band column, another renames landuse to land_use, a third widens population from int32 to int64, and a Shapefile round trip silently clips maintenance_schedule to maintenanc. None of these are geometry edits and none are attribute-value conflicts, yet any of them will break a naive merge, corrupt a downstream join, or throw a driver error at write time. This guide builds a reconciliation layer that treats the schema itself as a versioned artifact: fingerprint it, diff it, classify each change, merge under an explicit policy, and gate the merge on compatibility.

The techniques here sit one level of abstraction above attribute reconciliation for tabular spatial data, which resolves conflicting values once the columns already line up. Schema drift reconciliation is what you run first — it guarantees the columns line up at all. It also complements delta tracking algorithms for vector data: a delta engine can only diff features cleanly when both sides share a schema, so a fingerprint mismatch is the signal to run reconciliation before delta computation.

Prerequisites & Environment Setup

Schema reconciliation reads field definitions directly from the driver layer, so the toolchain is deliberately close to the metal — Fiona for schema introspection, GeoPandas for record handling, and pyarrow for typed columnar staging.

• Python 3.10+ with fiona>=1.9, geopandas>=0.14, pyarrow>=14, and shapely>=2.0
• GDAL 3.6+ available to Fiona (fiona.__gdal_version__ should report it)
• A canonical geometry type per layer decided up front (e.g. MultiPolygon for parcels) so promotion rules are unambiguous
• A field-naming convention that survives the Shapefile 10-character limit if Shapefile is anywhere in your pipeline (snake_case, ≤10 chars, no collisions after truncation)
• A schema registry file (schema/registry.json) recording the accepted fingerprint of each layer at each release tag
• UTC-aware datetime fields and an explicit width/precision for every numeric column, so int vs float and field-width drift are detectable

python -m venv .venv && source .venv/bin/activate
pip install "fiona>=1.9" "geopandas>=0.14" "pyarrow>=14" "shapely>=2.0"
# Confirm GDAL is wired through Fiona
python -c "import fiona; print(fiona.__version__, fiona.__gdal_version__)"

A schema in Fiona is a plain dict with two keys: geometry (a WKT-style type token such as "Polygon" or "3D MultiLineString") and properties, an ordered mapping of field name to a type token like "int:9", "float:24.15", or "str:80". Everything in this guide operates on that dict. The two companion deep-dives implement the heavy lifting: automating attribute schema migration with Fiona turns a diff into a field-level migration plan and applies it record by record, while reconciling geometry type changes across branches handles the harder case where the geometry token itself has drifted.

Core Algorithmic Patterns

1. Schema Fingerprinting

A fingerprint is a hash of the canonicalized schema. Canonicalization matters more than the hash function: field order, the exact spelling of type tokens, and geometry casing all vary between drivers, so you must normalize before hashing or two identical schemas will produce different fingerprints.

import hashlib, json
import fiona

def canonical_schema(schema: dict) -> dict:
    """Normalize a Fiona schema dict so equal schemas canonicalize identically."""
    props = {
        name.lower(): _canonical_type(typ)
        for name, typ in schema["properties"].items()
    }
    return {
        "geometry": _canonical_geom(schema["geometry"]),
        "properties": dict(sorted(props.items())),  # order-independent
    }

def _canonical_type(typ: str) -> str:
    # "float: 24.15" -> "float:24.15"; "Integer64" -> "int"
    base = typ.split(":")[0].strip().lower()
    width = typ.split(":")[1].strip() if ":" in typ else ""
    base = {"integer": "int", "integer64": "int", "string": "str"}.get(base, base)
    return f"{base}:{width}" if width else base

def _canonical_geom(geom) -> str:
    if geom is None:
        return "none"
    if isinstance(geom, (list, tuple)):        # driver may report a set of types
        return "|".join(sorted(g.lower() for g in geom))
    return str(geom).lower()

def fingerprint(schema: dict) -> str:
    canon = canonical_schema(schema)
    blob = json.dumps(canon, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()[:16]

with fiona.open("branches/main/parcels.gpkg") as src:
    print(fingerprint(src.schema))

Complexity: O© in the field count C. A fingerprint comparison across an N-layer repository is O(N) hash lookups — cheap enough to run as a pre-merge gate on every push.

2. Three-Way Schema Merge

Given the common-ancestor schema (base), the current branch (target), and the incoming branch (source), a three-way merge decides the reconciled field set the same way a text merge does: a field changed on only one side is taken from that side; a field changed on both sides is a conflict resolved by policy. Working at the schema level, the “change” is an add, drop, rename, or retype rather than an edited line.

3. Type-Widening Rules

Automated coercion is only ever allowed to move up a lattice of widening-safe conversions, never down. Encoding the lattice as a partial order lets the merge pick a common supertype deterministically.

# Each type maps to the set of types it can be safely widened INTO.
WIDEN_TO: dict[str, set[str]] = {
    "bool":  {"bool", "int", "float", "str"},
    "int":   {"int", "float", "str"},        # int -> float loses nothing up to 2**53
    "float": {"float", "str"},
    "date":  {"date", "datetime", "str"},
    "datetime": {"datetime", "str"},
    "str":   {"str"},
}

def widen(a: str, b: str) -> str | None:
    """Smallest common type a and b can both widen into, or None if incompatible."""
    common = WIDEN_TO[a] & WIDEN_TO[b]
    if not common:
        return None
    order = ["bool", "int", "float", "date", "datetime", "str"]
    return min(common, key=order.index)

The str fallback is intentional: any two scalar types can meet at str, but a merge that silently stringifies a numeric column is almost always wrong, so the policy layer treats a widen that resolves to str as an escalation rather than a clean auto-merge. Field width widens by taking the max (str:40 and str:80 reconcile to str:80), which is safe everywhere except Shapefile, where the 254-character cap and 10-character name limit are hard boundaries checked separately.

Production Workflow Implementation

The pipeline runs four stages — detect, classify, migrate, validate — with fingerprinting bracketing the whole run so the gate can prove the migrated layers converged.

Step 1 — Detect Drift

Read all three schemas and diff them into structured change sets. Keep the diff purely descriptive; classification is a separate concern.

from dataclasses import dataclass, field

@dataclass
class SchemaDiff:
    added:   dict[str, str] = field(default_factory=dict)   # name -> type
    removed: dict[str, str] = field(default_factory=dict)
    retyped: dict[str, tuple[str, str]] = field(default_factory=dict)  # name -> (old, new)
    geometry_changed: tuple[str, str] | None = None

def diff_schema(base: dict, other: dict) -> SchemaDiff:
    b = canonical_schema(base)
    o = canonical_schema(other)
    bp, op = b["properties"], o["properties"]
    d = SchemaDiff()
    for name, typ in op.items():
        if name not in bp:
            d.added[name] = typ
        elif bp[name] != typ:
            d.retyped[name] = (bp[name], typ)
    for name, typ in bp.items():
        if name not in op:
            d.removed[name] = typ
    if b["geometry"] != o["geometry"]:
        d.geometry_changed = (b["geometry"], o["geometry"])
    return d

A rename surfaces here as a simultaneous removed + added pair. Distinguishing a true rename from an unrelated drop-plus-add requires a value-similarity heuristic; the migration deep-dive resolves that ambiguity, so at detection time both are recorded verbatim.

Step 2 — Classify Each Change

Map every change onto a severity so the policy layer can decide what auto-merges and what escalates.

def classify(diff: SchemaDiff) -> dict[str, list]:
    buckets = {"safe": [], "widening": [], "breaking": []}
    for name, typ in diff.added.items():
        buckets["safe"].append(("add", name, typ))
    for name, (old, new) in diff.retyped.items():
        target = widen(old.split(":")[0], new.split(":")[0])
        if target is None or target == "str" and "str" not in (old, new):
            buckets["breaking"].append(("retype", name, old, new))
        else:
            buckets["widening"].append(("retype", name, old, new, target))
    for name, typ in diff.removed.items():
        buckets["breaking"].append(("drop", name, typ))   # additive-only: never auto-drop
    if diff.geometry_changed:
        buckets["breaking"].append(("geometry", *diff.geometry_changed))
    return buckets

Step 3 — Build the Reconciled Schema and Migrate

Under the additive-only policy the reconciled schema is the union of all fields, each at its widest safe type, with the canonical geometry. Breaking changes short-circuit the merge before any record is written.

def reconcile_schema(base: dict, target: dict, source: dict,
                     canonical_geometry: str) -> dict:
    dt = classify(diff_schema(base, target))
    ds = classify(diff_schema(base, source))
    breaking = dt["breaking"] + ds["breaking"]
    if breaking:
        raise SchemaGateError(f"{len(breaking)} breaking change(s) require review: {breaking}")

    props = dict(canonical_schema(base)["properties"])
    for bucket in (dt, ds):
        for kind, name, *rest in bucket["safe"] + bucket["widening"]:
            if kind == "add":
                props[name] = rest[0]
            elif kind == "retype":
                old, new, tgt = rest
                width = max(_width(old), _width(new))
                props[name] = f"{tgt}:{width}" if width else tgt
    return {"geometry": canonical_geometry, "properties": dict(sorted(props.items()))}

Once the reconciled schema exists, both branch layers are rewritten to conform to it — new columns backfilled with typed nulls, widened columns cast up. That per-record rewrite is exactly what automating attribute schema migration with Fiona implements end to end, and when the geometry token differs, reconciling geometry type changes across branches supplies the safe promotion rules that let a merge proceed.

Step 4 — Validate and Gate

Re-fingerprint the migrated layers. If they do not converge to the reconciled fingerprint, the migration was incomplete and the merge must fail closed.

def gate(migrated_target: dict, migrated_source: dict, reconciled: dict) -> None:
    fp = fingerprint(reconciled)
    for label, sch in (("target", migrated_target), ("source", migrated_source)):
        if fingerprint(sch) != fp:
            raise SchemaGateError(f"{label} did not converge to reconciled schema {fp}")

Code Reliability Patterns

Null typing on backfill. A new column added to old records must be filled with a typed null, not a bare None that the driver may coerce inconsistently. Materialize the fill value from the field type — 0/0.0 are wrong because they are legitimate data; use the driver’s null and record which features were backfilled in a _backfilled_fields audit column so the value can never be mistaken for a genuine measurement.

Range-checked coercion. Every cast is validated before it is committed. An int64 value exceeding 2**31 - 1 must not be written into an int32/int:9 Shapefile field; a float coerced to int must be integral within tolerance. Wrap coercion so an out-of-range value raises rather than truncating silently.

def safe_cast(value, from_t: str, to_t: str):
    if value is None:
        return None
    if from_t == "int" and to_t == "float":
        return float(value)
    if from_t == "float" and to_t == "int":
        if abs(value - round(value)) > 1e-9:
            raise ValueError(f"lossy float->int cast on {value!r}")
        return int(round(value))
    if to_t == "str":
        return str(value)
    return value

Idempotent migration. Running reconciliation twice must be a no-op the second time. Because the reconciled schema is a deterministic function of the three inputs, re-running produces the same fingerprint; guard the write path with if fingerprint(current) == fingerprint(reconciled): return so retries and CI re-runs never double-apply widening.

Registry-backed rollback. Record the accepted fingerprint per layer per tag in schema/registry.json. If a downstream consumer breaks, you can diff the current fingerprint against the last-known-good entry to pinpoint exactly which field drifted, and revert the layer to the prior schema without guessing.

Performance & Scale Considerations

Schema operations are cheap; record migration is where cost lives. Fingerprinting and diffing touch only the header, so they are effectively free even across a large repository — the table below is measured on a warm page cache over a 1.2 M-feature parcel layer.

Two levers matter at scale. First, gate on fingerprints, migrate lazily — only layers whose fingerprint changed since the last accepted registry entry need a record rewrite, which in incremental editing is typically a small fraction of the repository. Second, stage through pyarrow rather than casting cell-by-cell: read the layer into an Arrow table, apply widening casts as vectorized column operations, then write once. The columnar cast path roughly halves migration time versus a per-record Fiona loop and keeps memory bounded because Arrow columns are contiguous. For layers that will feed a diff engine afterward, aligning the schema first also makes delta tracking algorithms for vector data markedly faster, since a matched schema lets the delta pass compare feature rows positionally instead of reconciling columns on the fly.

Troubleshooting & Failure Modes

FAQ

What counts as schema drift in a vector layer?

Any change to the field definitions or the geometry declaration between two versions of the same layer: added or removed columns, renamed fields, changed data types, altered field widths or precision, changed coded-value enumerations, and a changed geometry type. Drift is orthogonal to row-level edits — it changes the shape of the table itself, not the values inside it, which is why it needs its own detection and reconciliation pass before value-level attribute reconciliation can run.

Why hash a schema instead of comparing dicts directly?

A fingerprint collapses a full schema comparison into a single equality check, so a CI gate can fast-path thousands of unchanged layers and only fall back to a structural diff when a hash mismatch appears. The hash also serves as a stable identifier you record in commit metadata and the schema registry, letting you detect historical drift without re-opening every file. Direct dict comparison still runs once a mismatch is found, to explain what drifted.

What is an additive-only reconciliation policy?

Additive-only permits new fields and safe type widening but forbids automatically dropping, renaming, or narrowing existing fields. New columns are backfilled with typed nulls in older records so no branch is retroactively invalidated. Destructive changes are never auto-applied; they are captured as breaking changes and routed to a migration script plus human review, which keeps the automated path safe by construction.

How do Shapefile field-width limits cause silent drift?

The Shapefile format truncates field names to 10 characters and enforces fixed field widths and a 254-character string cap. Writing a GeoPackage layer out to Shapefile can silently rename land_use_code to land_use_c and clip long text, so a later round trip appears as a rename plus a width change even though nobody edited the schema. Comparing the pre-write and post-write fingerprints exposes this immediately.

Can type coercion lose data during migration?

Yes, which is why only widening coercions run automatically. Narrowing float to int truncates, narrowing int64 to int32 overflows, and parsing strings to numbers fails on non-parseable cells. Reconciliation range-checks every cast and raises on any lossy conversion, so a merge can never silently corrupt attribute values — a narrowing request becomes an escalation, not a truncation.

Related

• Automating Attribute Schema Migration with Fiona — turn a schema diff into a field-level migration plan and apply it record by record
• Reconciling Geometry Type Changes Across Branches — detect and safely promote a drifted geometry type before merge
• Attribute Reconciliation for Tabular Spatial Data — resolve conflicting attribute values once the schema is aligned
• Delta Tracking Algorithms for Vector Data — compute feature-level diffs that depend on a matched schema
• Automated Conflict Detection in Merge Requests — wire schema gates into pre-merge validation

Back to Conflict Resolution & Team Synchronization Workflows