How to Tag and Release Versioned OpenStreetMap Extracts

Producing a deterministic, auditable OSM extract release requires pairing a semantic version tag with an immutable spatial snapshot and a machine-readable manifest — part of the broader release tagging strategies for spatial basemaps that govern how geospatial teams distribute reference data.

Concept & Context

OpenStreetMap data mutates continuously through thousands of daily edits. A .osm.pbf file downloaded today will differ from one downloaded tomorrow, even for the same geographic boundary. Without explicit versioning, downstream GIS pipelines — routing engines, tile servers, environmental models — silently diverge as teams pull extracts at different moments.

The solution is dual identification: every extract carries a human-readable semantic version (vMAJOR.MINOR.PATCH) and a machine-readable temporal anchor embedded as the OSM replication sequence number (osmosis_replication_sequence_number) and replication timestamp (osmosis_replication_timestamp). These two fields together let any consumer reconstruct the exact state of the planet file at extraction time, independent of wall-clock dates.

This approach builds directly on the branching and merge strategies for spatial datasets that govern how feature work is isolated before a release is cut. Teams following those conventions should create a dedicated release candidate branch, run validation, and only tag once CI gates pass — the same discipline applied here at the artifact level.

The diagram below shows the full lifecycle from a raw planet mirror to a versioned, verified artifact in the registry.

OSM extract release pipeline Six stages from left to right: OSM Planet Mirror, osmium extract, VERSION.json manifest, SHA-256 checksum, Git tag, Artifact Registry. Arrows connect each stage in sequence. OSM Planet Mirror osmium extract VERSION.json manifest SHA-256 checksum Git annotated tag + LFS push step 1 step 2 step 3 step 4 steps 5–6 step 7 Registry

Core Algorithmic Pipeline

Step 1 — Resolve the semantic version and replication anchor

Before touching any files, decide which version component increments:

  • MAJOR: extraction boundary (--polygon), coordinate reference system (EPSG:4326 → other), or output format (.osm.pbf → .geojson) changed.
  • MINOR: filter expressions, tag transformations, or attribute enrichment rules changed.
  • PATCH: upstream OSM replication feed advanced; extraction logic is unchanged.

Record the current replication sequence number before the extract runs:

osmium fileinfo -g header.option.osmosis_replication_sequence_number source.osm.pbf

Store this integer in a variable; it becomes the canonical temporal anchor in the manifest.

Step 2 — Extract the regional PBF snapshot

Use osmium extract with a --polygon GeoJSON or Osmosis poly file to enforce a deterministic geographic boundary. Always specify --strategy=complete-ways to prevent split ways at the boundary edge, which would fail topology checks downstream:

osmium extract \
  --polygon boundary/city-of-nairobi.geojson \
  --strategy=complete-ways \
  --output releases/nairobi-v1.4.2.osm.pbf \
  planet/africa-latest.osm.pbf

Tag the boundary file itself in version control. If the polygon changes between releases, that is a MAJOR increment regardless of how small the geometry shift appears.

Step 3 — Generate the VERSION.json manifest

The manifest is the machine-readable contract consumed by downstream pipelines. It must travel alongside the artifact in every distribution channel.

Step 4 — Compute the SHA-256 checksum

Write a .sha256 sidecar file in the GNU sha256sum format so consumers can run sha256sum -c without parsing JSON:

sha256sum releases/nairobi-v1.4.2.osm.pbf > releases/nairobi-v1.4.2.sha256

Step 5 — Commit metadata and annotate the Git tag

Stage only the manifest and checksum — never the binary PBF, which belongs in Git LFS or object storage. An annotated tag (not a lightweight tag) stores the tagger identity and timestamp in the object database, satisfying audit trail requirements described in security boundaries for spatial repositories:

git add releases/nairobi-v1.4.2.json releases/nairobi-v1.4.2.sha256
git commit -m "release: nairobi v1.4.2"
git tag -a v1.4.2 -m "OSM extract nairobi v1.4.2 — PATCH refresh, seq 4182901"
git push origin main --follow-tags

Step 6 — Push binary artifacts to remote storage

For repositories using Git LFS:

git lfs track "*.osm.pbf"
git add .gitattributes
git commit -m "chore: track PBF files with Git LFS"
git lfs push --all origin

For S3-compatible object storage, sync with the structured {region}-{version} naming convention so pipelines can discover artifacts programmatically:

aws s3 sync ./releases/ s3://spatial-artifacts/osm-extracts/ \
  --exclude "*" \
  --include "nairobi-v1.4.2.*" \
  --acl bucket-owner-full-control

Working Implementation

The script below integrates all six steps. It expects osmium, git, and git lfs in $PATH and a pre-downloaded source PBF. Outputs land in a ./releases/ directory: the extracted PBF, a VERSION.json manifest, and a .sha256 sidecar.

#!/usr/bin/env python3
"""
Automated OSM extract versioning pipeline.

Usage:
    python tag_release.py <region> <source.pbf> <boundary.geojson> <vX.Y.Z>

Produces:
    releases/{region}-{version}.osm.pbf      — extracted snapshot
    releases/{region}-{version}.json         — VERSION.json manifest
    releases/{region}-{version}.sha256       — SHA-256 sidecar
"""

import hashlib
import json
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path


def run(cmd: list[str], cwd: str | None = None) -> str:
    """Run a subprocess and return stripped stdout; raise on non-zero exit."""
    result = subprocess.run(cmd, capture_output=True, text=True, check=True, cwd=cwd)
    return result.stdout.strip()


def sha256_file(path: Path) -> str:
    """Compute SHA-256 hex digest in 8 KiB chunks (streaming — safe for large PBFs)."""
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(8192), b""):
            h.update(chunk)
    return h.hexdigest()


def osm_replication_meta(pbf_path: Path) -> dict[str, str]:
    """
    Read OSM replication timestamp and sequence number embedded in the PBF header.
    Both fields are written by Geofabrik and the osmosis replication toolchain.
    Returns empty strings on failure so the manifest is still valid.
    """
    def _get(field: str) -> str:
        try:
            return run(["osmium", "fileinfo", "-g", f"header.option.{field}", str(pbf_path)])
        except subprocess.CalledProcessError:
            return ""

    return {
        "osm_replication_timestamp": _get("osmosis_replication_timestamp"),
        "osm_replication_sequence_number": _get("osmosis_replication_sequence_number"),
    }


def tag_and_release(
    region: str,
    source_pbf: str,
    boundary_geojson: str,
    version: str,
    output_dir: str = "./releases",
) -> None:
    out = Path(output_dir)
    out.mkdir(parents=True, exist_ok=True)

    target_pbf  = out / f"{region}-{version}.osm.pbf"
    meta_path   = out / f"{region}-{version}.json"
    sha_path    = out / f"{region}-{version}.sha256"

    source = Path(source_pbf)
    if not source.is_file():
        raise FileNotFoundError(f"Source PBF not found: {source_pbf}")
    if not Path(boundary_geojson).is_file():
        raise FileNotFoundError(f"Boundary GeoJSON not found: {boundary_geojson}")

    # --- Step 2: extract regional snapshot ---------------------------------
    if not target_pbf.exists():
        print(f"Extracting {region} → {target_pbf.name} …")
        run([
            "osmium", "extract",
            "--polygon", boundary_geojson,
            "--strategy=complete-ways",
            "--output", str(target_pbf),
            str(source),
        ])
    else:
        print(f"Using existing extract: {target_pbf.name}")

    # --- Step 3: VERSION.json manifest -------------------------------------
    repl = osm_replication_meta(target_pbf)
    manifest = {
        "region": region,
        "version": version,
        "format": "osm.pbf",
        "file": target_pbf.name,
        "sha256": "",                        # filled in after checksum step
        "extracted_at": datetime.now(timezone.utc).isoformat(),
        "boundary": str(Path(boundary_geojson).name),
        "osm_replication_timestamp": repl["osm_replication_timestamp"],
        "osm_replication_sequence_number": repl["osm_replication_sequence_number"],
        "generator": "osmium-tool",
        "license": "ODbL 1.0",
    }

    # --- Step 4: SHA-256 checksum ------------------------------------------
    print("Computing SHA-256 checksum …")
    checksum = sha256_file(target_pbf)
    manifest["sha256"] = checksum
    sha_path.write_text(f"{checksum}  {target_pbf.name}\n", encoding="utf-8")
    meta_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
    print(f"Manifest written: {meta_path.name}")

    # --- Steps 5–6: Git tag + LFS push ------------------------------------
    try:
        run(["git", "add", str(meta_path), str(sha_path)])
        run(["git", "commit", "-m", f"release: {region} {version}"])
        run(["git", "tag", "-a", version,
             "-m", f"OSM extract {region} {version} seq={repl['osm_replication_sequence_number']}"])
        run(["git", "push", "origin", "main", "--follow-tags"])
        print(f"Tagged and pushed: {version}")
    except subprocess.CalledProcessError as exc:
        print(f"Git operation failed:\n{exc.stderr}", file=sys.stderr)
        sys.exit(1)


if __name__ == "__main__":
    if len(sys.argv) != 5:
        print("Usage: python tag_release.py <region> <source.pbf> <boundary.geojson> <vX.Y.Z>")
        sys.exit(1)
    tag_and_release(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4])

Validation & Output Verification

After the pipeline completes, verify the release before promoting it to downstream consumers.

1. Checksum integrity

cd releases/
sha256sum -c nairobi-v1.4.2.sha256
# expected: nairobi-v1.4.2.osm.pbf: OK

2. Manifest freshness check

Parse VERSION.json to confirm the replication timestamp falls within your permitted staleness window (typically 24–72 hours for tile servers, shorter for routing engines):

import json
from datetime import datetime, timezone

manifest = json.loads(open("releases/nairobi-v1.4.2.json").read())
extracted = datetime.fromisoformat(manifest["extracted_at"])
age_hours = (datetime.now(timezone.utc) - extracted).total_seconds() / 3600
assert age_hours < 72, f"Extract is {age_hours:.1f} h old — exceeds freshness threshold"
print(f"Freshness OK: {age_hours:.1f} h old, seq {manifest['osm_replication_sequence_number']}")

3. Geometry and feature-count sanity check

Run ogrinfo on the PBF to confirm node and way counts are within expected bounds. A count drop of more than 5 % compared with the previous release warrants manual review before publication:

ogrinfo -al -so releases/nairobi-v1.4.2.osm.pbf lines
# inspect: Feature Count, Extent, Layer SRS

4. Git tag verification

Confirm the annotated tag is present on the remote and references the correct commit:

git ls-remote --tags origin | grep v1.4.2
git show v1.4.2 --stat

Failure Modes

  • osmium fileinfo returns empty replication fields — the source PBF was produced by a tool that does not write the osmosis_replication_* header options (e.g., a custom Overpass export). Populate the fields manually from the Overpass API response timestamp or the Geofabrik download page.

  • SHA-256 mismatch on download — the artifact was re-uploaded or overwritten after tagging. Treat the tag as invalid, re-extract from the same source sequence, and re-tag with an incremented PATCH version.

  • git lfs push fails with “batch request too large” — the LFS server has a per-batch object-count limit. Push with --batch-size 1 to send each object individually: git lfs push --all --batch-size 1 origin.

  • Downstream pipeline loads wrong extract due to ambiguous naming — two regions share a common prefix (e.g., nairobi and nairobi-north). Enforce exact-match lookups against the region field inside VERSION.json rather than relying on filename prefix matching.