Cloud-Native Spatial Formats for Versioned Pipelines

Choosing a storage format decides how expensive your version history will be — part of Choosing Formats & Tools for Spatial Data Versioning. Cloud-native formats such as Cloud Optimized GeoTIFF (COG), Zarr, FlatGeobuf, and GeoParquet split a dataset into internally addressable pieces, which turns an expensive whole-file re-upload into a cheap re-upload of only the pieces that actually changed.

This guide explains what “cloud-native” means in practice, why internal chunking makes chunk-level versioning nearly free, and how to build a manifest workflow that commits only changed chunks. It closes with reliability patterns, a chunk-size tuning benchmark, and a troubleshooting table you can lift into your own pipeline.

Prerequisites & Environment Setup

Before you build a chunk-versioned pipeline, confirm the toolchain and the invariants that keep chunk hashes stable across versions:

• Python 3.10+ with rasterio, xarray, zarr, numcodecs, pyarrow, and geopandas
• GDAL 3.6+ with the COG and FlatGeobuf drivers (gdalinfo --formats | grep -E "COG|FlatGeobuf")
• A content-addressable object store (S3/GCS/MinIO) plus DVC or a plain manifest committed to Git
• A fixed chunk grid, compression codec, and codec level pinned in a project config file
• A single project CRS (e.g. EPSG:4326 for delivery tiles, a projected CRS such as EPSG:32633 for analysis) established before the first write
• Deterministic writer settings — no timestamps in headers, sorted keys — so identical data yields identical bytes
• 8 GB+ RAM for hashing large stores; NVMe scratch space for staging chunk blobs

python -m pip install "rasterio>=1.3" "xarray>=2024.1" "zarr>=2.17" \
    numcodecs "pyarrow>=15" geopandas "dvc[s3]>=3.0"
gdalinfo --version
gdalinfo --formats | grep -E "COG|FlatGeobuf|Parquet"

These formats descend from the same lineage as pointer synchronization for raster datasets: the heavy binary lives in object storage, and Git tracks only a small manifest. The difference here is granularity — instead of one hash per multi-gigabyte file, you keep one hash per internal chunk.

Core Algorithmic Patterns

1. HTTP range reads and internal tiling

“Cloud-native” is a read-time property first. A COG places its image data in fixed tiles (typically 256×256 or 512×512) and appends reduced-resolution overviews, with a leading header that maps every tile to a byte offset. A client issues an HTTP range request for exactly the tiles that intersect a viewport, ignoring the rest of a file that may be tens of gigabytes. Zarr generalises this to N dimensions: each chunk is a separately addressable object keyed by its grid position (for example temp/2/4/1). FlatGeobuf packs a static R-tree at the head of the file so a reader can binary-search a bounding box and range-read only the matching feature records. GeoParquet splits data into row groups, and within each row group each column is a separately compressed chunk with min/max statistics for predicate pushdown.

The versioning payoff is a direct consequence: the same internal boundary that lets a reader fetch one tile also lets a writer replace one tile. The addressable unit for reads is the addressable unit for change.

2. Chunk-addressable versioning

Content-addressable storage keys each blob by the hash of its bytes. Apply it at chunk granularity and version comparison becomes set arithmetic. Given version A and version B of a store, compute the hash of every chunk in each, then:

• Chunks whose hash appears in both A and B are unchanged — never re-uploaded.
• Chunks in B but not A are new or modified — uploaded once.
• Chunks in A but not B are deleted — dropped from the new manifest.

Complexity: hashing is O(n) in total bytes, but the transfer cost is O(changed bytes), not O(total bytes). For an edit touching k of N chunks, upload volume scales with k while the manifest scales with N. This is the entire economic argument for chunked formats in a version-control pipeline.

3. Per-chunk hashing and the manifest

The manifest is a small, sortable, text-serialisable record: one row per chunk holding its logical key, byte size, and content hash. Committed to Git, it is diffable — a reviewer sees exactly which tiles moved between two commits. The table below summarises where the addressable unit lives in each format.

Production Workflow Implementation

The goal is a repeatable loop: build a chunk-aligned store, hash every chunk into a manifest, diff against the last committed manifest, and upload only the differences. The diagram shows the full path.

Step 1: Build a chunk-aligned store

Write the raster as a tiled COG (or an equivalently chunked Zarr store) with the chunk grid, compression, and predictor pinned. These settings must never change between versions, or every chunk re-encodes.

import rasterio
from rasterio.shutil import copy as rio_copy

COG_PROFILE = {
    "driver": "COG",
    "blocksize": 512,          # fixed tile size — pin this forever
    "compress": "DEFLATE",     # deterministic codec
    "predictor": 2,            # horizontal differencing
    "overview_resampling": "average",
    "BIGTIFF": "IF_SAFER",
}

def build_cog(src_path: str, dst_path: str) -> None:
    """Rewrite a raster as a tiled COG with fixed, version-stable settings."""
    with rasterio.open(src_path) as src:
        rio_copy(src, dst_path, **COG_PROFILE)

Step 2: Enumerate and hash each internal chunk

For a COG, iterate the block windows the driver already defined; for Zarr, iterate the chunk keys under the array directory. Hash each chunk’s raw bytes with a streaming digest.

import hashlib
import rasterio

def hash_cog_tiles(path: str, algo: str = "sha256") -> dict[str, str]:
    """Return {block_key: hex_digest} for every internal tile of a COG."""
    manifest: dict[str, str] = {}
    with rasterio.open(path) as src:
        for band in range(1, src.count + 1):
            for (row, col), window in src.block_windows(band):
                data = src.read(band, window=window)
                h = hashlib.new(algo)
                h.update(data.tobytes())
                manifest[f"b{band}/{row}/{col}"] = h.hexdigest()
    return manifest

Step 3: Write the chunk manifest

Serialise the manifest to sorted JSON so identical stores always produce byte-identical manifests — a precondition for a clean Git diff.

import json
from pathlib import Path

def write_manifest(manifest: dict[str, str], out_path: str) -> None:
    """Persist a chunk manifest as sorted, deterministic JSON."""
    payload = {"algo": "sha256", "chunks": dict(sorted(manifest.items()))}
    Path(out_path).write_text(json.dumps(payload, indent=2, sort_keys=True))

Step 4: Diff against the previous version

Load the manifest committed with the previous version and compute the changed, added, and removed keys.

import json
from pathlib import Path

def diff_manifests(old_path: str, new: dict[str, str]) -> dict[str, set]:
    """Compare a new chunk manifest against the last committed one."""
    old = {}
    if Path(old_path).exists():
        old = json.loads(Path(old_path).read_text())["chunks"]
    old_keys, new_keys = set(old), set(new)
    changed = {k for k in old_keys & new_keys if old[k] != new[k]}
    return {
        "added": new_keys - old_keys,
        "removed": old_keys - new_keys,
        "changed": changed,
    }

Step 5: Commit and upload only changed chunks

Upload the blobs for added | changed, then commit the new manifest. Unchanged chunks are already in object storage under the same content hash, so they are never sent again.

import subprocess

def push_changed(diff: dict[str, set], bucket: str, blob_for) -> int:
    """Upload only added/changed chunk blobs; return the count pushed."""
    to_send = diff["added"] | diff["changed"]
    for key in sorted(to_send):
        local_path, remote_key = blob_for(key)  # your key → path mapping
        subprocess.run(
            ["aws", "s3", "cp", local_path, f"s3://{bucket}/{remote_key}"],
            check=True,
        )
    return len(to_send)

git add store.manifest.json
git commit -m "Version raster: 12 tiles changed, 748 unchanged (skipped upload)"

Code Reliability Patterns

Pin every encoding parameter. A single change to blocksize, compress, or predictor re-encodes all chunks and produces an all-new manifest, so guard the profile with an assertion before writing:

def assert_stable_profile(profile: dict, expected: dict) -> None:
    for key in ("blocksize", "compress", "predictor"):
        if profile.get(key) != expected.get(key):
            raise ValueError(
                f"Encoding drift on '{key}': {profile.get(key)} != {expected.get(key)} "
                "— this would re-hash every chunk and break deduplication."
            )

Verify before you delete. Before dropping a chunk that the diff marks as removed, confirm no other committed manifest still references its hash. Content-addressed deletion is safe only when the hash is unreferenced everywhere.

Snap edits to the chunk grid. When you know a bounding box was edited, restrict re-hashing to the intersecting chunks rather than rescanning the whole store. This keeps the hashing cost proportional to the edited region and mirrors the tolerance-snapping discipline used elsewhere in versioned spatial pipelines.

Fail closed on partial uploads. If the upload loop aborts midway, do not commit the manifest — a committed manifest that references a chunk missing from storage produces a silent read failure on the next checkout. Upload first, verify object presence, then commit.

Performance & Scale Considerations

Chunk size is the master dial. Smaller chunks localise edits (fewer wasted bytes per change) but inflate manifest size and per-request overhead; larger chunks cut overhead but re-upload more per edit. The table below shows a representative single-tile edit on a 40 GB, four-band COG (reference hardware: 16-core VM, NVMe, 10 Gbps).

The sweet spot for most editing workflows is a 512×512 or 1024×1024 tile: manifests stay a few megabytes, per-edit uploads stay small, and range reads remain efficient. For deep dives on how N-dimensional chunk shape steers which chunks turn dirty, see chunking Zarr arrays for incremental raster commits. For vector delivery, versioning FlatGeobuf streams for web delivery covers producing deterministic, diff-stable feature files.

Parallelise hashing. Chunk hashing is embarrassingly parallel — fan out block windows across a ProcessPoolExecutor to saturate cores. Batch small uploads. When a diff touches many tiny chunks, pack them into a single multipart upload rather than issuing thousands of PUT requests.

Troubleshooting & Failure Modes

FAQ

What makes a spatial format “cloud-native”?

A cloud-native format supports HTTP range reads and stores data in internally addressable pieces — tiles, chunks, or row groups — alongside an index or overview pyramid. A reader can fetch a bounding box or a single band without downloading the whole file. COG, Zarr, FlatGeobuf, and GeoParquet all satisfy this, and that same internal addressing is exactly what makes chunk-level versioning cheap.

Why does internal chunking make versioning cheaper?

Because a small edit only rewrites the chunks it touches. If you hash each chunk independently and store a manifest of those hashes, a new version re-hashes every chunk but re-uploads only the ones whose hash changed. A 200 GB Zarr store where one tile changed pushes a few megabytes, not 200 GB — the transfer cost tracks the edited region, not the dataset size.

How do I keep chunk hashes stable across writes?

Fix the chunk grid, compression codec, and codec level up front, and never let the writer re-tile between versions. Any change to chunk shape, block size, or compression re-encodes every chunk and produces all-new hashes, defeating deduplication. Pin these parameters in a config committed alongside the data and assert them before every write.

Should I version the whole file or the chunk manifest?

Commit the manifest — a small text file listing chunk keys and hashes — to Git, and store the chunk blobs in content-addressed object storage. This mirrors pointer synchronization for raster datasets: Git holds cheap, auditable lineage while heavy binary chunks live in a bucket and are fetched on demand.

What chunk size should I choose?

Aim for 1–8 MB compressed per chunk. Smaller chunks give finer-grained deltas but inflate manifest size and per-request overhead; larger chunks reduce overhead but re-upload more data per edit. Tune to your edit locality: point edits favour smaller chunks, while full-scene refreshes tolerate larger ones.

Related

• Chunking Zarr Arrays for Incremental Raster Commits — how chunk shape determines which chunks turn dirty, and how to hash only those
• Versioning FlatGeobuf Streams for Web Delivery — deterministic FlatGeobuf output and tile versioning for web maps
• Pointer Synchronization for Raster Datasets — the pointer/manifest pattern this workflow builds on
• GeoParquet vs GeoPackage vs Shapefile for Versioned Workflows — choosing a vector format for diff-friendly commits
• Large File Handling in DVC for GIS — backend routing and cache layout for the blobs your manifest points at

Back to Choosing Formats & Tools for Spatial Data Versioning