Chunking Zarr Arrays for Incremental Raster Commits

The chunk shape you pick when you write a Zarr array silently decides how expensive every future commit will be, because an edit re-hashes exactly the chunks its footprint overlaps. This page shows how to choose chunk dimensions, detect the dirty chunks after an edit, and commit only those — part of Cloud-Native Spatial Formats for Versioned Pipelines.

Concept & Context

Zarr stores an N-dimensional array as a grid of independently compressed chunks, each written as one object whose key encodes its position in the grid (for example 0.3.1 for a chunk at band 0, row block 3, column block 1). Array metadata in .zarray records the shape, the chunk shape, and the compressor. This is the same internal addressability that makes the format readable over HTTP range requests, and it is what makes incremental versioning possible: the chunk is both the read unit and the change unit.

The consequence for version control is blunt. When you modify a region of the array, Zarr rewrites every chunk that region overlaps — not the pixels, the whole chunk. A ten-pixel edit that straddles a chunk boundary rewrites two chunks; the same edit centred inside one chunk rewrites one. So the number of dirty objects per commit is governed by the relationship between your chunk shape and your edit footprint, not by how many pixels you changed. Choosing chunk dimensions is therefore a versioning decision, not only a performance one, and it complements the file-level pointer synchronization for raster datasets approach used for monolithic GeoTIFFs.

Core Chunking Pipeline

  1. Profile your edit footprint. Determine how edits typically arrive: scattered single-pixel corrections, contiguous windows, or full time-step refreshes. This footprint is the template the chunk shape should match.
  2. Choose a chunk shape. Size chunks so a typical edit touches as few as possible while keeping each chunk around 1–8 MB compressed. For time series, chunk the time axis to 1 so each step is a self-contained object.
  3. Write with a fixed grid. Persist the array with an explicit chunks= argument and a pinned compressor. Record these in a config so no later write silently re-chunks.
  4. Map edits to chunk keys. Given the index bounds of an edited region, compute the set of chunk grid indices it overlaps via integer division of the bounds by the chunk shape.
  5. Hash and diff. Hash the bytes of each candidate chunk and compare against the baseline manifest to confirm which are genuinely dirty (an overlapping chunk whose values did not change keeps its hash).
  6. Commit the dirty chunks. Upload only the changed chunk objects and record the new manifest.

Working Implementation

The script below writes a chunked Zarr array, applies an edit to a window, identifies exactly which chunk keys that window intersects, re-hashes only those, and reports the dirty set to commit. It uses xarray, zarr, and numpy.

#!/usr/bin/env python3
"""
zarr_incremental_commit.py

Write a chunked Zarr array, apply a windowed edit, and determine which
chunks turned dirty so only those are re-hashed and committed.

Requirements: xarray>=2024.1, zarr>=2.17, numpy
"""

import hashlib
import numpy as np
import xarray as xr
import zarr


CHUNK_SHAPE = (512, 512)      # pin this: re-chunking rehashes everything
STORE = "elevation.zarr"
VAR = "elevation"


def write_array(store: str) -> None:
    """Create a 2048x2048 float32 array with a fixed 512x512 chunk grid."""
    data = np.random.default_rng(0).random((2048, 2048), dtype="float32")
    da = xr.DataArray(data, dims=("y", "x"), name=VAR)
    da.to_dataset().to_zarr(
        store,
        mode="w",
        encoding={VAR: {"chunks": CHUNK_SHAPE, "compressor": zarr.Blosc("zstd", 5)}},
    )


def chunk_keys_for_window(y0: int, y1: int, x0: int, x1: int,
                          chunks=CHUNK_SHAPE) -> set[tuple[int, int]]:
    """Return the (row_block, col_block) chunk indices a pixel window overlaps.

    An edit dirties every chunk its footprint touches, so we floor-divide the
    inclusive window bounds by the chunk shape to enumerate the grid cells.
    """
    cy, cx = chunks
    rows = range(y0 // cy, (y1 - 1) // cy + 1)
    cols = range(x0 // cx, (x1 - 1) // cx + 1)
    return {(r, c) for r in rows for c in cols}


def hash_chunk(store: str, row: int, col: int, algo: str = "sha256") -> str:
    """Read one raw Zarr chunk object and return its content hash."""
    arr = zarr.open(store, mode="r")[VAR]
    cy, cx = arr.chunks
    block = arr[row * cy:(row + 1) * cy, col * cx:(col + 1) * cx]
    h = hashlib.new(algo)
    h.update(np.ascontiguousarray(block).tobytes())
    return h.hexdigest()


def baseline_manifest(store: str) -> dict[tuple[int, int], str]:
    """Hash every chunk once to establish the pre-edit baseline."""
    arr = zarr.open(store, mode="r")[VAR]
    cy, cx = arr.chunks
    n_rows = -(-arr.shape[0] // cy)   # ceil division
    n_cols = -(-arr.shape[1] // cx)
    return {(r, c): hash_chunk(store, r, c)
            for r in range(n_rows) for c in range(n_cols)}


def apply_edit(store: str, window: tuple[int, int, int, int]) -> None:
    """Write a constant patch into a pixel window, re-chunking nothing."""
    y0, y1, x0, x1 = window
    ds = xr.open_zarr(store)
    ds[VAR][y0:y1, x0:x1] = 42.0
    region = {"y": slice(y0, y1), "x": slice(x0, x1)}
    ds[[VAR]].isel(region).to_zarr(store, region=region)


def main() -> None:
    write_array(STORE)
    baseline = baseline_manifest(STORE)

    window = (600, 640, 1000, 1050)          # small edit straddling a boundary
    apply_edit(STORE, window)

    # Only chunks the window overlaps can possibly be dirty:
    candidates = chunk_keys_for_window(*window)
    dirty = []
    for (r, c) in sorted(candidates):
        new_hash = hash_chunk(STORE, r, c)
        if new_hash != baseline[(r, c)]:
            dirty.append((r, c))

    total = len(baseline)
    print(f"Chunks overlapping edit: {len(candidates)} of {total}")
    print(f"Dirty chunks to commit : {len(dirty)} -> {dirty}")
    # Commit step: upload only `dirty` chunk objects + updated manifest.


if __name__ == "__main__":
    main()

The key move is chunk_keys_for_window: instead of re-hashing all sixteen chunks of the array, it floor-divides the edit bounds by the chunk shape to enumerate only the chunks the window can touch, then hashes just those. On the 512-pixel grid, the window (600, 640, 1000, 1050) falls entirely inside row block 1 and column block 1, so a single chunk is a candidate and a single chunk turns dirty — out of sixteen total. Widen the window across a boundary and the candidate set grows accordingly, which is exactly the behaviour chunk shape controls.

Validation & Output Verification

Confirm the chunk grid, the dirty count, and the on-disk object layout:

# Inspect chunk shape and compressor recorded in metadata
python -c "import zarr; a=zarr.open('elevation.zarr')['elevation']; \
print('shape', a.shape, 'chunks', a.chunks, 'compressor', a.compressor)"

# List the raw chunk objects on disk (one file per grid cell)
ls elevation.zarr/elevation | head

# Expect the script to report a dirty count far smaller than total chunks
python zarr_incremental_commit.py

Assert the invariants in code so a regression fails loudly rather than silently re-uploading the whole array:

import zarr
arr = zarr.open("elevation.zarr")["elevation"]
assert arr.chunks == (512, 512), "chunk grid drifted — would rehash everything"
assert arr.shape == (2048, 2048)

If the reported dirty count ever approaches the total chunk count for a small edit, the grid or compressor changed between versions and every chunk re-encoded — treat that as a build failure.

Failure Modes

  • Symptom: a tiny edit reports the entire array as dirty. Root cause: the array was rewritten with a different chunks= or compressor, re-encoding every chunk. Fix: pin chunk shape and compressor in the write config and assert them before writing.
  • Symptom: dirty count is larger than expected for a compact edit. Root cause: the edit window straddles chunk boundaries, dirtying neighbours. Fix: align edit windows to the chunk grid, or shrink the chunk shape over hot regions.
  • Symptom: to_zarr(region=...) raises a boundary error. Root cause: the region slice is not aligned to the chunk grid for a region write. Fix: expand the region to whole-chunk bounds, or write the full variable if the edit is dense.
  • Symptom: two identical edits produce different chunk hashes. Root cause: a non-deterministic compressor setting or differing codec level between runs. Fix: fix the Blosc codec and level and record them in metadata so bytes are reproducible.

Back to Cloud-Native Spatial Formats for Versioned Pipelines