Versioning Controlled Vocabularies and Domain Codes

A layer’s land_use column is meaningless without the code list it was assigned under — and code lists change, which is how a value that meant one thing in 2019 quietly means another today. This page is a focused companion to schema drift reconciliation for spatial layers.

Concept & Context

Schema drift is usually discussed in terms of columns: fields added, removed, retyped. The subtler drift is in values. A column called land_use keeps its name and its type while the set of permitted codes underneath it changes — a category is split into two, another retired, a third redefined without changing its code.

Nothing in a schema check catches that. Both layers have a land_use text column; both contain plausible codes. The merge succeeds, and the merged layer contains values from two vocabularies with no marker saying which is which. Downstream, an area summary by category quietly double-counts a split and undercounts its successors.

Treating the vocabulary as a versioned artifact in its own right fixes it. Each layer declares which version its values belong to, transitions between versions are published as explicit mappings, and a validation gate rejects any value that does not exist in the declared version.

Core Algorithmic Pipeline

  1. Publish the vocabulary as a versioned artifact with a stable identifier, a version, and a content hash.
  2. Declare the version in each layer’s metadata, not in a comment or a wiki.
  3. Publish a mapping per transition, marking each old code as renamed, split, merged, retired or unchanged.
  4. Migrate deliberately, refusing to guess where a mapping is ambiguous.
  5. Validate on every commit that a layer’s values exist in the version it declares.
What a version transition does to each code A graph of codes across two vocabulary versions: one code passes through unchanged, one splits into two successors that no rule can choose between, and one is retired with no successor at all. v2 v3 AGR AGR RESIDENTIAL RES_LOW RES_HIGH The split has no automatic answer: choosing the commoner successor manufactures observations nobody made.

Working Implementation

"""Versioned controlled vocabularies with explicit, auditable transitions."""
from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class Vocabulary:
    identifier: str
    version: str
    codes: dict[str, str]              # code -> label

    @classmethod
    def load(cls, path: str) -> "Vocabulary":
        doc = json.loads(Path(path).read_text(encoding="utf-8"))
        return cls(doc["identifier"], doc["version"],
                   {c["code"]: c["label"] for c in doc["codes"]})

    @property
    def digest(self) -> str:
        payload = json.dumps(sorted(self.codes.items()), separators=(",", ":"))
        return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16]


@dataclass(frozen=True)
class Transition:
    """How each code in `from_version` maps into `to_version`."""
    identifier: str
    from_version: str
    to_version: str
    rules: dict[str, dict]

    @classmethod
    def load(cls, path: str) -> "Transition":
        doc = json.loads(Path(path).read_text(encoding="utf-8"))
        return cls(doc["identifier"], doc["from"], doc["to"],
                   {r["code"]: r for r in doc["rules"]})

    def apply(self, code: str) -> tuple[str | None, str]:
        """Return (new code or None, reason). None means a human must decide."""
        rule = self.rules.get(code)
        if rule is None:
            return None, "no rule for this code — the transition is incomplete"

        kind = rule["kind"]
        if kind in ("unchanged", "renamed"):
            return rule["to"], kind
        if kind == "merged":
            return rule["to"], "merged into a broader category"
        if kind == "split":
            # Deliberately unresolvable: the successor depends on facts the code
            # does not carry. Guessing here invents observations nobody made.
            return None, (f"split into {rule['to']} — needs re-survey or an "
                          "explicit per-feature decision")
        if kind == "retired":
            return None, f"retired: {rule.get('note', 'no successor')}"
        raise ValueError(f"unknown transition kind {kind!r} for {code!r}")


def validate_layer(gdf, column: str, vocab: Vocabulary) -> list[dict]:
    """Rows whose value is not in the declared vocabulary version."""
    bad = []
    for idx, value in gdf[column].items():
        if value is None:
            continue
        if str(value) not in vocab.codes:
            bad.append({"row": int(idx), "value": str(value)})
    return bad


def migrate_layer(gdf, column: str, transition: Transition):
    """Apply a transition, returning the migrated frame and the unresolved rows."""
    migrated = gdf.copy()
    unresolved = []
    for idx, value in gdf[column].items():
        if value is None:
            continue
        new_code, reason = transition.apply(str(value))
        if new_code is None:
            unresolved.append({"row": int(idx), "value": str(value), "reason": reason})
        else:
            migrated.at[idx, column] = new_code
    return migrated, unresolved
// vocabularies/land-use/v3.json
{
  "identifier": "gdv:land-use",
  "version": "3",
  "codes": [
    {"code": "RES_LOW",  "label": "Residential — low density"},
    {"code": "RES_HIGH", "label": "Residential — high density"},
    {"code": "IND_LIGHT","label": "Industrial — light"},
    {"code": "IND_HEAVY","label": "Industrial — heavy"},
    {"code": "AGR",      "label": "Agricultural"}
  ]
}
// vocabularies/land-use/v2-to-v3.json
{
  "identifier": "gdv:land-use",
  "from": "2", "to": "3",
  "rules": [
    {"code": "AGR", "kind": "unchanged", "to": "AGR"},
    {"code": "RESIDENTIAL", "kind": "split", "to": ["RES_LOW", "RES_HIGH"],
     "note": "density threshold introduced in v3; needs a dwelling count"},
    {"code": "INDUSTRIAL", "kind": "split", "to": ["IND_LIGHT", "IND_HEAVY"]},
    {"code": "MIXED", "kind": "retired", "note": "no successor; re-survey required"}
  ]
}

The layer declares its version where the pipeline can read it:

// data/parcels.gpkg.meta.json
{
  "vocabularies": {
    "land_use": {"identifier": "gdv:land-use", "version": "3",
                 "digest": "b4f21c9d8a3e77f0"}
  }
}
The five transition kinds, and which ones a migration may apply A grid of transition kinds — unchanged, renamed, merged, split and retired — against whether a migration can resolve each automatically and what it must do when it cannot. Automatic? What the migration does unchanged yes copies the code through renamed yes applies the declared new code merged yes, with loss records that detail was folded away split no reports the rows for re-survey retired no reports them with the retirement note Two of five rows require a human, and a migration that hides that is manufacturing data.

Validation & Output Verification

# Every value must exist in the declared version
import json, geopandas as gpd
from vocabulary import Vocabulary, validate_layer

meta = json.load(open("data/parcels.gpkg.meta.json"))
decl = meta["vocabularies"]["land_use"]
vocab = Vocabulary.load(f"vocabularies/land-use/v{decl['version']}.json")

assert vocab.digest == decl["digest"], (
    "the layer declares a vocabulary digest that no longer matches the artifact — "
    "the vocabulary was edited in place instead of being versioned"
)

bad = validate_layer(gpd.read_file("data/parcels.gpkg"), "land_use", vocab)
assert not bad, f"{len(bad)} row(s) hold codes absent from v{vocab.version}: {bad[:5]}"
print(f"all values valid in {vocab.identifier} v{vocab.version}")
# A transition must cover every code in the source version
python - <<'PY'
from vocabulary import Vocabulary, Transition
old = Vocabulary.load("vocabularies/land-use/v2.json")
t = Transition.load("vocabularies/land-use/v2-to-v3.json")
missing = sorted(set(old.codes) - set(t.rules))
assert not missing, f"transition has no rule for: {missing}"
print(f"transition covers all {len(old.codes)} codes in v2")
PY

# Two layers must not be merged across vocabulary versions
jq -r '.vocabularies.land_use.version' data/*.gpkg.meta.json | sort -u | wc -l
# expected: 1

The digest check is the one that catches the real-world failure: somebody edits v3.json in place to add a code rather than publishing v4, and every layer claiming v3 now claims something that no longer exists.

Four checks that keep a vocabulary honest Four validation steps: every layer declares a vocabulary version, the declared digest still matches the published artifact, every value exists in that version, and no merge crosses two versions. 1 Every layer declares a version in metadata the pipeline reads 2 The digest still matches catches an edit-in-place of the code list 3 Every value exists in it and the failure names the rows 4 No merge across versions totals that do not reconcile start here The digest check is the one that catches a v3 that quietly gained a code without becoming v4.

Failure Modes

  • A merged layer holds two vocabularies — symptom: category totals that do not reconcile. Root cause: no version declared, so the merge could not detect the mismatch. Fix: declare the version per layer and block merges across versions.

  • A split was resolved automatically — symptom: a suspiciously even distribution across new categories. Root cause: migration guessed a successor. Fix: leave splits unresolved and route them to re-survey; the migration should report them, not solve them.

  • The vocabulary was edited in place — symptom: the digest check fails, or worse, passes because nobody recorded one. Root cause: treating the code list as a document rather than a versioned artifact. Fix: publish a new version for any change, however small.

  • A transition is incomplete — symptom: migration leaves values untouched with no report. Root cause: codes with no rule. Fix: fail the transition when any source code lacks a rule, as the coverage check does.

Back to Schema Drift Reconciliation for Spatial Layers