Setting Up Secure Access Controls for Versioned Shapefiles

Enforce role-bound, integrity-checked permissions that treat every shapefile group as an indivisible unit — from directory ACLs through Git branch gates to cloud IAM policies. Part of Security Boundaries in Spatial Repositories.

Concept & Context

A shapefile is not a single file. The ESRI specification mandates at least three tightly coupled components — .shp (geometry), .shx (shape index), and .dbf (attribute table) — and a production dataset almost always adds .prj (coordinate reference system), .cpg (character encoding), and optional spatial indexes (.sbn/.sbx). Every component shares the same stem; losing or locking any one of them corrupts the entire dataset.

This coupling creates a problem that the broader security boundaries in spatial repositories model must account for explicitly: standard version control systems treat files as independent objects. Git can accept a .shp commit while a pre-receive hook rejects the corresponding .dbf on a policy violation, leaving the repository in a broken state. Unlike formats with native row-level locking or embedded metadata, shapefiles have no internal mechanism to signal partial integrity. The access control layer must therefore enforce atomicity externally, across three synchronized enforcement surfaces: the storage filesystem, the version control layer, and the CI/CD pipeline.

Three-Layer Access Control Stack for Versioned Shapefiles Storage layer at the bottom (POSIX/NTFS ACLs, cloud bucket policies), version control layer in the middle (branch protections, CODEOWNERS, pre-commit hooks), and CI/CD layer at the top (integrity validation, role routing, audit logs). Arrows show enforcement flowing upward through each layer. Storage Layer POSIX / NTFS ACLs · Cloud bucket prefix policies · Object-level IAM tags Version Control Layer Branch protections · CODEOWNERS · Pre-commit integrity hooks CI / CD Layer Integrity validation · Role-based merge routing · Immutable audit logs

Core Algorithmic Pipeline

Step 1 — Inventory and classify every shapefile group. Scan the repository and tag each stem with a security tier (public, internal, restricted). Record the classification in a sidecar .json manifest alongside the shapefile group. This manifest becomes the authoritative input for all downstream policy checks.

Step 2 — Apply directory-level ACLs. On Linux/macOS, use setfacl to grant named-group access to the containing directory rather than individual files. On Windows NTFS, apply DACL entries at folder level with inheritance. In cloud storage (S3, GCS, Azure Blob), write prefix-scoped bucket policies that map the shapefile stem’s parent prefix to an IAM role. Never rely on per-extension permissions — if a user can read parcels.shp they must not be able to read parcels.dbf via a separate, weaker policy.

Step 3 — Configure CODEOWNERS and branch protections. Place a CODEOWNERS file at the repository root and assign directory ownership to named GIS leads (e.g., /data/admin_boundaries/ @gis-admin-team). Enable required reviewers and required status checks in your Git host’s branch protection settings. Any pull request touching a protected directory must pass peer review and the integrity validator before it can merge.

Step 4 — Install the pre-commit integrity validator. Register the Python script below as a Git pre-commit hook (chmod +x .git/hooks/pre-commit). The hook intercepts every staged commit, walks the staged file list to find shapefile stems, and verifies that all mandatory extensions are present and unmodified before allowing the commit to proceed.

Step 5 — Enforce tag-based IAM in cloud object storage. Tag each uploaded shapefile group with metadata keys (classification, owner, project). Write IAM condition expressions that require the calling principal’s tags to match the object’s tags before allowing PutObject or DeleteObject. Enable object versioning so historical states are retained, and use object lock or legal holds for any dataset subject to regulatory retention requirements.

Step 6 — Validate, audit, and monitor. Run a nightly CI job that compares the on-disk classification manifests against the storage backend’s tag values and the Git CODEOWNERS entries. Alert on any drift. Write all policy decisions to an append-only audit log. This closing loop is what turns a one-time configuration into an ongoing, auditable security posture.

Working Implementation

The script below runs as a Git pre-commit hook or a CI pipeline step. It validates shapefile completeness, enforces contributor role membership, and exits non-zero on any violation, blocking the commit before it reaches the repository.

#!/usr/bin/env python3
"""
pre-commit hook / CI gate: shapefile integrity + role-based access validator.
Install: cp validate_shapefiles.py .git/hooks/pre-commit && chmod +x .git/hooks/pre-commit
"""

import os
import subprocess
import sys
from pathlib import Path
from typing import Dict, List, Set

# All extensions that must travel together under the same stem.
# .prj is mandatory here — an ambiguous CRS is a security surface (see notes).
REQUIRED_EXTENSIONS: Set[str] = {".shp", ".shx", ".dbf", ".prj"}

# Optional extensions that, when present, must also be consistent.
OPTIONAL_EXTENSIONS: Set[str] = {".cpg", ".sbn", ".sbx", ".xml"}

# Role → allowed directory prefixes mapping (extend from CI secrets or LDAP in production).
ROLE_DIRECTORY_MAP: Dict[str, Set[str]] = {
    "gis-admin":    {"data/admin_boundaries", "data/restricted"},
    "data-steward": {"data/"},
    "contributor":  {"data/public", "data/internal"},
    "ci-bot":       {"data/"},  # CI service account — broad but audited
}


def get_staged_files() -> List[Path]:
    """Return the list of files currently staged for commit."""
    result = subprocess.run(
        ["git", "diff", "--cached", "--name-only", "--diff-filter=ACMR"],
        capture_output=True,
        text=True,
        check=True,
    )
    return [Path(p) for p in result.stdout.splitlines() if p.strip()]


def find_shapefile_stems(staged: List[Path]) -> Set[str]:
    """Extract unique stems from staged shapefile components."""
    stems = set()
    all_extensions = REQUIRED_EXTENSIONS | OPTIONAL_EXTENSIONS
    for p in staged:
        if p.suffix.lower() in all_extensions:
            # Store stem as 'dir/name' (without extension) for grouping.
            stems.add(str(p.parent / p.stem))
    return stems


def validate_integrity(stem: str) -> bool:
    """
    Confirm all mandatory components exist on disk for the given stem.
    Does NOT rely solely on staged files — checks the working tree so
    a selective add cannot hide a missing component.
    """
    base = Path(stem)
    found = {p.suffix.lower() for p in base.parent.glob(f"{base.name}.*")}
    missing = REQUIRED_EXTENSIONS - found

    if missing:
        print(
            f"  [FAIL] Integrity: '{base.name}' is missing {', '.join(sorted(missing))}. "
            f"All components must be committed together."
        )
        return False
    return True


def resolve_current_role() -> str:
    """
    Map the current Git author to a role.
    In production, replace this with an LDAP query, IAM token claim,
    or a lookup against a secrets-manager-backed role table.
    """
    author = os.environ.get("GIT_AUTHOR_NAME", os.environ.get("USER", "unknown"))
    role_env = os.environ.get("GIS_ROLE", "")  # Set by CI from OIDC claims

    # Simple heuristic: CI bots carry a known env var.
    if "CI" in os.environ or author.endswith("-bot"):
        return "ci-bot"
    if role_env in ROLE_DIRECTORY_MAP:
        return role_env
    # Default to most restricted role for unknown authors.
    return "contributor"


def validate_role_access(stem: str, role: str) -> bool:
    """
    Confirm the current role is authorised to modify files under the stem's directory.
    """
    allowed_prefixes = ROLE_DIRECTORY_MAP.get(role, set())
    stem_path = str(Path(stem).parent) + "/"

    for prefix in allowed_prefixes:
        if stem_path.startswith(prefix) or prefix == "data/":
            return True

    print(
        f"  [FAIL] Access: role '{role}' is not authorised to modify '{stem_path}'. "
        f"Request access from a gis-admin or data-steward."
    )
    return False


def check_gitattributes_binary_flag(stem: str) -> bool:
    """
    Warn (non-fatal) if .gitattributes does not mark shapefiles as binary.
    Line-ending normalisation corrupts .dbf headers and .shp byte offsets.
    """
    gitattributes = Path(".gitattributes")
    if not gitattributes.exists():
        print(
            f"  [WARN] .gitattributes not found. Add '*.shp binary' and '*.dbf binary' "
            f"to prevent CRLF corruption of binary components."
        )
    return True  # Warning only — do not block the commit


def main() -> int:
    print("=== Shapefile access-control pre-commit hook ===")

    try:
        staged = get_staged_files()
    except subprocess.CalledProcessError as exc:
        print(f"  [ERROR] Could not read staged files: {exc}")
        return 1

    stems = find_shapefile_stems(staged)
    if not stems:
        # No shapefiles staged — nothing to validate.
        return 0

    role = resolve_current_role()
    print(f"  Detected role: '{role}'")
    print(f"  Shapefile stems to validate: {len(stems)}")

    passed = True
    for stem in sorted(stems):
        print(f"\n  Checking: {stem}")
        if not validate_integrity(stem):
            passed = False
        if not validate_role_access(stem, role):
            passed = False
        check_gitattributes_binary_flag(stem)

    if passed:
        print("\n  [PASS] All shapefile integrity and access checks passed.")
        return 0
    else:
        print("\n  [BLOCK] Commit rejected. Fix the errors above and re-stage.")
        return 1


if __name__ == "__main__":
    sys.exit(main())

Validation & Output Verification

After installing the hook, verify it fires correctly before relying on it in production.

Trigger a deliberate failure: Stage only parcels.shp without its .dbf and attempt a commit. The hook should print [FAIL] Integrity: 'parcels' is missing .dbf, .prj and exit non-zero, blocking the commit.

git add data/public/parcels.shp
git commit -m "test: should be blocked"
# Expected: hook exits 1 with integrity failure message

Verify a clean pass: Stage the complete group together.

git add data/public/parcels.shp data/public/parcels.shx \
        data/public/parcels.dbf data/public/parcels.prj
git commit -m "feat: add parcels shapefile"
# Expected: hook exits 0, commit proceeds

Confirm the .gitattributes binary flag is set:

grep -E '\.(shp|dbf|shx)' .gitattributes
# Expected output: *.shp binary  *.dbf binary  *.shx binary

Audit staged role decisions: Set GIS_ROLE=contributor in your shell and attempt to commit a file under data/restricted/. The hook should reject with an access-denied message, confirming the role map is enforced.

Failure Modes

  • .shp committed without .dbf — root cause: developer used git add *.shp with a glob that excluded other extensions; the hook catches this if installed, but if the hook was bypassed with --no-verify, restore from the previous commit and re-add the complete group. Never run --no-verify on spatial data repositories.

  • Shapefile opened in QGIS or ArcGIS during commit — root cause: GIS desktop software creates .shp.lock files that the hook may misread as a valid extension or that interfere with staging; configure the hook to ignore .lock suffixes and remind contributors to close the file before committing.

  • Cloud IAM policy denies the CI service account — root cause: the ci-bot role tag was not propagated to the object’s owner tag at upload time; verify that the upload step in the pipeline explicitly sets all required metadata tags before the policy evaluation runs.

  • CRS missing from .prj file — root cause: some GIS export workflows omit .prj when the dataset uses a non-standard or custom CRS; the hook blocks the commit because .prj is in REQUIRED_EXTENSIONS. Fix by explicitly writing the CRS using ogr2ogr -a_srs EPSG:4326 before staging, then verify with ogrinfo -al -so yourfile.shp | grep 'AUTHORITY'.