Querying a Parcel Layer as of a Past Commit

Answering “what did this parcel look like in April” against a repository means resolving a date to a commit, materialising the layer at that revision, and querying it — a small pipeline that is easy to get subtly wrong. This page is a focused companion to temporal versioning and time-travel queries.

Concept & Context

Every versioned repository already contains its own history, so the obvious way to answer a historical question is to go and look. That instinct is right, and the mechanics are less obvious than they appear, because three things have to line up: the date has to resolve to a specific commit, that commit’s pointer files have to resolve to artifacts the object store still holds, and the query has to run against a materialisation that is genuinely the historical state rather than a partially updated working tree.

It is worth being precise about what this answers. A commit-based as-of query returns the state the repository recorded by a date. If the April survey was committed in May, an as-of query for late April will not see it. That is transaction time, and for many questions — what did we publish, what did the analyst see, what was the basis of that decision — it is exactly the right answer. When the question is about the ground rather than the record, the bitemporal model is the tool, and this technique is the fallback that works before that model exists.

The approach costs nothing to adopt. There is no schema change, no migration, and no new store — it uses history that is already there. What it costs is query latency on the first request for a revision, which is why the caching step is not optional in anything user-facing.

Core Algorithmic Pipeline

  1. Resolve the instant to a commit. Take the last commit on the publication branch whose committer date is at or before the target instant. Resolve on the publication branch specifically, not on whatever branch is checked out, or the answer depends on the caller’s working state.
  2. Verify the pointer resolves. Read the artifact’s pointer file at that commit and confirm the object store still holds the hash it names, before doing any work.
  3. Materialise into a revision-scoped directory. Fetch into a path keyed by the commit hash, never into the working tree, so a historical query cannot disturb current work or be disturbed by it.
  4. Query the materialised layer. From here it is an ordinary spatial query against an ordinary file.
  5. Retain by commit hash. The materialisation for an immutable commit is itself immutable, so it can be cached until space pressure evicts it.
From a date to an answer A chain of four stages: an instant resolves to a commit on the publication branch, the commit resolves to a pointer, the pointer resolves to an artifact hash in the object store, and the materialised layer answers the query. Instant 2025-04-01 rev-list Commit last before that date git show Pointer the .dvc file at that commit dvc get Artifact hash in the object store Each arrow can fail, and each failure means something different — which is why none of them should fall back.

Working Implementation

"""As-of query against a DVC-tracked layer, resolved through the commit graph."""
import hashlib
import json
import subprocess
from datetime import datetime, timezone
from pathlib import Path

import geopandas as gpd

CACHE_ROOT = Path("/var/cache/gdv-asof")
PUBLICATION_BRANCH = "main"


def resolve_commit(instant: datetime, branch: str = PUBLICATION_BRANCH) -> str:
    """Last commit on `branch` recorded at or before `instant`.

    Committer date, not author date: author date travels with a cherry-picked
    or rebased commit and would resolve to a revision the branch never held.
    """
    iso = instant.astimezone(timezone.utc).isoformat()
    sha = subprocess.check_output(
        ["git", "rev-list", "-1", f"--before={iso}", branch], text=True
    ).strip()
    if not sha:
        raise LookupError(f"{branch} has no commit at or before {iso}")
    return sha


def pointer_at(commit: str, pointer_path: str) -> dict:
    """Read a .dvc pointer as it stood at `commit` without checking anything out."""
    blob = subprocess.check_output(
        ["git", "show", f"{commit}:{pointer_path}"], text=True
    )
    import yaml
    return yaml.safe_load(blob)["outs"][0]


def materialise(commit: str, pointer_path: str, artifact_name: str) -> Path:
    """Fetch the artifact as of `commit` into a revision-scoped cache directory."""
    target_dir = CACHE_ROOT / commit
    target = target_dir / artifact_name
    if target.exists():
        return target                                    # immutable commit, immutable result

    out = pointer_at(commit, pointer_path)
    target_dir.mkdir(parents=True, exist_ok=True)

    fetched = subprocess.run(
        ["dvc", "get", ".", out["path"], "--rev", commit, "-o", str(target)],
        capture_output=True, text=True,
    )
    if fetched.returncode != 0:
        raise RuntimeError(
            f"artifact for {out['path']} at {commit[:12]} is not retrievable: "
            f"{fetched.stderr.strip()}"
        )

    digest = hashlib.md5(target.read_bytes()).hexdigest()
    if digest != out["md5"]:
        target.unlink()
        raise RuntimeError(
            f"hash mismatch at {commit[:12]}: pointer says {out['md5']}, "
            f"fetched bytes hash to {digest}"
        )

    (target_dir / "provenance.json").write_text(json.dumps({
        "commit": commit,
        "pointer": pointer_path,
        "md5": out["md5"],
        "materialised_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    }, indent=2))
    return target


def parcels_as_of(instant: datetime, bbox=None) -> gpd.GeoDataFrame:
    """Return the parcel layer as the repository recorded it at `instant`."""
    commit = resolve_commit(instant)
    path = materialise(commit, "data/parcels.gpkg.dvc", "parcels.gpkg")
    gdf = gpd.read_file(path, bbox=bbox)
    gdf.attrs["as_of_commit"] = commit
    gdf.attrs["as_of_requested"] = instant.isoformat()
    return gdf


if __name__ == "__main__":
    target = datetime(2025, 4, 1, tzinfo=timezone.utc)
    layer = parcels_as_of(target, bbox=(4321000, 3210000, 4326000, 3215000))
    print(f"{len(layer)} parcels as of {target:%Y-%m-%d} "
          f"(commit {layer.attrs['as_of_commit'][:12]})")

Two details carry most of the reliability. The hash comparison after fetching means a corrupted or substituted object is caught here rather than in the analysis that follows. And attaching as_of_commit to the returned frame means every downstream result can name the revision it came from — which matters enormously when two analysts produce different numbers for the same date.

Why the second query for a date costs nothing A sequence showing a first as-of query resolving a commit, missing the cache, fetching from the object store and verifying the hash, followed by a second query for a different spatial extent on the same date that is served entirely from the revision-scoped cache. Analyst Resolver Cache Object store parcels as of 2025-04-01 commit 4f2a91c present? miss fetch + verify the hash the only expensive step same date, different extent hit — a commit is immutable Caching by commit hash is safe precisely because a commit can never mean something else later.

Validation & Output Verification

Confirm the resolution is doing what you think before trusting any answer built on it:

# The commit the date resolves to, and what it says about itself
git rev-list -1 --before=2025-04-01T00:00:00+00:00 main \
  | xargs -I{} git show -s --format='%H%n%cI%n%s' {}

# The pointer as it stood at that commit
git show "$(git rev-list -1 --before=2025-04-01T00:00:00+00:00 main):data/parcels.gpkg.dvc"

# Whether the object store still holds it
dvc status --cloud --rev "$(git rev-list -1 --before=2025-04-01T00:00:00+00:00 main)"

Then check the answer is stable and the cache is honest:

# Two runs of the same query must return an identical feature count and commit
python -c "
from asof import parcels_as_of
from datetime import datetime, timezone
t = datetime(2025, 4, 1, tzinfo=timezone.utc)
a, b = parcels_as_of(t), parcels_as_of(t)
assert len(a) == len(b) and a.attrs['as_of_commit'] == b.attrs['as_of_commit']
print('stable:', len(a), 'features at', a.attrs['as_of_commit'][:12])
"

# A cache directory must never exist without its provenance file
find /var/cache/gdv-asof -mindepth 1 -maxdepth 1 -type d \
  ! -exec test -e '{}/provenance.json' \; -print

The last check catches a half-written cache entry left behind by an interrupted fetch, which otherwise returns a truncated file that opens without complaint and reports the wrong feature count.

Failure Modes

  • The query answers with today’s data — symptom: an as-of query for last year returns the current parcel count. Root cause: materialising into the working tree, so the current file was read. Fix: fetch into a revision-scoped directory and never resolve paths relative to the working tree.

  • Two analysts get different answers for one date — symptom: the same as-of question produces different feature counts. Root cause: resolution ran against whatever branch each caller had checked out. Fix: resolve against the publication branch explicitly, as the implementation above does.

  • A historical query silently returns the nearest available revision — symptom: results for a date the object store cannot actually serve. Root cause: a fallback on fetch failure. Fix: fail loudly; a wrong answer to a historical question is worse than an error, particularly where the question came from an audit.

  • The first query of the day takes minutes — symptom: unpredictable latency on historical requests. Root cause: cold cache plus a large artifact. Fix: pre-warm the cache for revisions that get queried regularly — release tags especially — on a schedule.

Three ways an as-of query answers the wrong question A grid of three failures with cause and fix: today's data returned for a historical date, two analysts disagreeing about the same date, and a silent substitution of the nearest retrievable revision. Root cause Fix Returns today's data materialised into the working tree fetch into a revision-scoped directory Two analysts, two answers resolved against the caller's branch resolve against the publication branch explicitly Nearest revision substituted a fallback on fetch failure fail loudly — a wrong historical answer is worse than none All three return a plausible layer, which is what makes them expensive.

Back to Temporal Versioning and Time-Travel Queries