Redacting Sensitive Attributes Before a Public Release

The public layer is derived from the internal one, and everything that makes redaction hard follows from that: the two share identifiers, precision and structure, and each of those is a route back to what was meant to stay private. This page is a focused companion to security boundaries in spatial repositories.

Concept & Context

Redaction failures are rarely dramatic. Nobody publishes a column called owner_name. What gets published is a column added last quarter that nobody reviewed, an internal identifier that joins straight back to the private layer, or a point location precise enough to identify the property it sits on.

All three have the same structural cause: the public layer is produced by removing things from the internal one, and removal is a process that fails open. A column added after the removal list was written survives. An identifier kept “so consumers can track features across releases” reconnects the two datasets. Coordinates published at survey precision identify a building even when every attribute is stripped.

The fix is to invert the default. Build the public layer by selecting what may be published rather than by deleting what may not, derive identifiers rather than passing them through, and treat geometric precision as an attribute subject to the same policy as any other.

Core Algorithmic Pipeline

1. Declare an allow-list of columns, with a required disposition for each: publish as-is, generalise, or derive.
2. Derive the published identifier as a keyed hash with a per-release key.
3. Apply geometric generalisation where precision itself is identifying, using aggregation with a minimum count per unit.
4. Assert the result against the policy — unknown columns, residual values, precision floor, minimum counts.
5. Record the release with the policy version and the key identifier, so a later question can be answered without re-deriving anything.

Working Implementation

"""Produce a publishable layer from an internal one, failing closed."""
from __future__ import annotations

import hashlib
import hmac
import json
import secrets
from pathlib import Path

import geopandas as gpd
import yaml
from shapely import set_precision


def load_policy(path: str = "config/publication-policy.yml") -> dict:
    return yaml.safe_load(Path(path).read_text())


def derived_id(value: str, key: bytes) -> str:
    """Stable within a release, unlinkable across releases and to the source."""
    return hmac.new(key, str(value).encode("utf-8"), hashlib.sha256).hexdigest()[:20]


def redact(source: str, policy: dict, release_key: bytes) -> gpd.GeoDataFrame:
    gdf = gpd.read_file(source)

    allowed = policy["columns"]
    unknown = [c for c in gdf.columns
               if c not in allowed and c != gdf.geometry.name]
    if unknown:
        raise ValueError(
            "columns present in the source but absent from the publication policy: "
            f"{sorted(unknown)}. Add a disposition for each before releasing."
        )

    out = gpd.GeoDataFrame(geometry=gdf.geometry, crs=gdf.crs)
    for name, rule in allowed.items():
        if name not in gdf.columns:
            raise ValueError(f"policy names column {name!r} which the source lacks")

        disposition = rule["publish"]
        if disposition == "as_is":
            out[name] = gdf[name]
        elif disposition == "derive_id":
            out[rule.get("as", name)] = gdf[name].map(
                lambda v: derived_id(v, release_key)
            )
        elif disposition == "bucket":
            out[name] = _bucket(gdf[name], rule["buckets"])
        elif disposition == "drop":
            continue
        else:
            raise ValueError(f"unknown disposition {disposition!r} for {name!r}")

    grid = policy["geometry"]["precision_m"]
    out["geometry"] = out.geometry.apply(lambda g: set_precision(g, grid))

    return out


def _bucket(series, buckets: list[dict]):
    """Replace a continuous value with the band it falls in."""
    def band(value):
        for b in buckets:
            if value is not None and b["from"] <= value < b["to"]:
                return b["label"]
        return None
    return series.map(band)


def aggregate_points(gdf: gpd.GeoDataFrame, units: str, min_count: int) -> gpd.GeoDataFrame:
    """Replace identifying points with counts per published unit.

    Units holding fewer than min_count features are suppressed entirely rather
    than published with a small count, because a count of one is a location.
    """
    unit_gdf = gpd.read_file(units).to_crs(gdf.crs)
    joined = gpd.sjoin(gdf, unit_gdf[["unit_id", "geometry"]], predicate="within")
    counts = joined.groupby("unit_id").size().rename("feature_count").reset_index()

    merged = unit_gdf.merge(counts, on="unit_id", how="left")
    merged["feature_count"] = merged["feature_count"].fillna(0).astype(int)

    suppressed = merged[(merged["feature_count"] > 0) &
                        (merged["feature_count"] < min_count)]
    merged.loc[suppressed.index, "feature_count"] = None
    merged["suppressed"] = merged.index.isin(suppressed.index)
    return merged


def publish(source: str, out_path: str, policy_path: str = "config/publication-policy.yml"):
    policy = load_policy(policy_path)
    key = secrets.token_bytes(32)                 # a fresh key per release

    redacted = redact(source, policy, key)
    redacted.to_file(out_path, driver="GPKG")

    Path(out_path).with_suffix(".release.json").write_text(json.dumps({
        "policy_version": policy["version"],
        "key_id": hashlib.sha256(key).hexdigest()[:16],
        "columns_published": sorted(redacted.columns),
        "geometry_precision_m": policy["geometry"]["precision_m"],
    }, indent=2))
    # The key itself goes to the secret store, never to the repository.
    return redacted

# config/publication-policy.yml
version: 3
columns:
  parcel_uid:   {publish: derive_id, as: public_ref}
  land_use:     {publish: as_is}
  area_m2:      {publish: bucket, buckets:
                   [{from: 0,    to: 500,   label: "<500"},
                    {from: 500,  to: 2000,  label: "500-2000"},
                    {from: 2000, to: 1e9,   label: ">2000"}]}
  owner_ref:    {publish: drop}
  survey_notes: {publish: drop}
geometry:
  precision_m: 1.0

Every column in the source must appear here, including the ones being dropped. That requirement is what makes a newly added column stop the release rather than ride along with it.

Validation & Output Verification

# The published layer must not contain anything the policy did not authorise
import geopandas as gpd, json, yaml

published = gpd.read_file("release/parcels_public.gpkg")
policy = yaml.safe_load(open("config/publication-policy.yml"))
record = json.load(open("release/parcels_public.release.json"))

expected = {r.get("as", name) for name, r in policy["columns"].items()
            if r["publish"] != "drop"} | {"geometry"}
assert set(published.columns) == expected, set(published.columns) ^ expected

# No value may match a dropped source column
internal = gpd.read_file("data/parcels.gpkg")
for dropped in [n for n, r in policy["columns"].items() if r["publish"] == "drop"]:
    leaked = set(internal[dropped].dropna().astype(str)) & set(
        published.astype(str).values.ravel()
    )
    assert not leaked, f"{dropped} values present in the published layer: {list(leaked)[:3]}"

# The published identifier must not be joinable to the internal one
assert not (set(published["public_ref"]) & set(internal["parcel_uid"].astype(str)))

# Geometry precision must be at or coarser than the policy floor
import numpy as np
coords = np.concatenate([np.array(g.exterior.coords) for g in published.geometry])
residual = np.abs(coords - np.round(coords / record["geometry_precision_m"])
                  * record["geometry_precision_m"]).max()
assert residual < 1e-9, f"geometry finer than the policy floor by {residual}"
print("publication policy satisfied")

Run this as a required gate on the release pipeline, not as a script somebody remembers. A redaction check that is optional is a redaction check that will be skipped on the release that mattered.

Failure Modes

• A new column is published — symptom: an internal field appears in a public release. Root cause: a deny-list, which cannot know about columns added later. Fix: an allow-list that fails on any unrecognised column, as above.

• The public layer joins back to the internal one — symptom: a consumer reconstructs private attributes. Root cause: the internal identifier was published for convenience. Fix: publish a keyed hash with a per-release key, and keep the key out of the repository.

• Aggregation still identifies — symptom: a unit with a count of one. Root cause: no minimum-count suppression. Fix: suppress units below the threshold entirely rather than publishing a small count.

• Precision reintroduced downstream — symptom: a published service serves coordinates finer than the release. Root cause: the service reads the internal layer and applies its own filter. Fix: serve the published artifact, and give the service no access to the internal one — the tiering in the parent guide exists for exactly this.

Related

• Security Boundaries in Spatial Repositories — the parent guide and the tier model this implements
• Setting Up Secure Access Controls for Versioned Shapefiles — keeping the internal layer unreachable from the publication path
• Provenance and Lineage Tracking for Spatial Pipelines — recording that the redaction stage ran, with which policy version
• Release Tagging Strategies for Spatial Basemaps — where the release record and key identifier are published

Back to Security Boundaries in Spatial Repositories