Understanding Pointer Files in GeoGit vs DVC

Pointer files in geospatial version control replace binary raster payloads with lightweight content-addressable references — the mechanism that makes it practical to version multi-gigabyte GeoTIFFs and LiDAR tiles inside Git. This page compares the GeoGig (the modern, actively maintained successor to GeoGit) and DVC pointer architectures in precise detail. Parent context: Pointer Synchronization for Raster Datasets.

Concept & Context

Git’s object store is optimized for small text blobs. A 2 GB multi-band satellite scene committed as a raw binary bloats the pack file, inflates every git clone, and forces full-file deltas on every subsequent change. Pointer files solve this by committing a few-hundred-byte descriptor to Git while routing the actual binary to a content-addressed remote store — an approach documented in the broader geospatial data versioning fundamentals for any binary-heavy spatial asset.

Both GeoGig and DVC use this pattern but with fundamentally different philosophies. GeoGig extends Git’s native object model, natively parsing geometry, coordinate reference systems (EPSG:4326, EPSG:3857), and attribute schemas — meaning its pointer layer is tightly coupled to spatial semantics. DVC is framework-agnostic: it treats every tracked file as an opaque binary, generating a YAML pointer that any Git-enabled tool can read, with the remote backend swappable at any time without rewriting the pointer itself.

The choice between the two determines your entire pointer synchronization for raster datasets strategy: GeoGig if your workflow centres on spatial diffs and OGC tooling; DVC if it centres on Python pipelines, cloud storage flexibility, or machine-learning integration with rasterio or xarray.

GeoGig vs DVC pointer file architecture Two parallel data-flow diagrams. Left side shows GeoGig routing raster binaries through Git LFS pointer files to an LFS server. Right side shows DVC generating .dvc YAML files committed to Git while pushing binaries independently to S3/GCS/Azure remotes. GeoGig + Git LFS DVC imagery.tif (2 GB raster) geogig add version https://git-lfs… oid sha256:abcdef… (committed to Git) Git repo LFS server Spatial index: geometry / CRS diffs native imagery.tif (2 GB raster) dvc add imagery.tif.dvc md5: a1b2c3d4… (committed to Git) Git repo S3 / GCS / Azure Remote swappable; files treated as opaque

Core Algorithmic Pipeline

GeoGig + Git LFS pointer flow

  1. Threshold detection. When geogig add processes a raster, GeoGig evaluates blob size against a configurable threshold (default lfs.threshold = 100MB in .geogig/config). Files below the threshold are stored as native GeoGig objects with geometry parsing; files above are handed off to the Git LFS client.

  2. Pointer generation. The Git LFS clean filter hashes the binary with SHA-256 and writes a three-line text pointer committed to Git in place of the binary:

    version https://git-lfs.github.com/spec/v1
oid sha256:abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890
size 2147483648

  3. Binary upload. The LFS smudge filter transfers the binary to the configured LFS endpoint (lfs.url in .git/config). Authentication is handled via the Git credential helper or a GIT_LFS_TOKEN environment variable — never hardcoded in repository configuration.

  4. Spatial index maintenance. For vector layers and small rasters parsed natively, GeoGig maintains a spatial index enabling geogig diff --spatial and bounding-box queries against historical versions. LFS-backed blobs appear in the index as un-parsed references and cannot participate in geometry-aware diffs.

  5. Checkout reconciliation. On geogig checkout or git checkout, the LFS smudge filter intercepts pointer blobs and fetches the binary from the LFS server, placing the full file in the working tree. If the LFS endpoint is unreachable, the pointer file is left in place and operations on the raster will silently fail.

DVC pointer flow

  1. Checksum computation. dvc add imagery.tif computes an MD5 (or --hash sha256) digest of the file and writes a YAML pointer alongside it:

    outs:
- md5: a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6
  path: imagery.tif
  size: 2147483648
  isexec: false

  2. Local cache placement. DVC moves (or hard-links) the binary into .dvc/cache/ using a two-level directory structure keyed by the first two characters of the checksum: .dvc/cache/a1/b2c3d4…. Subsequent modifications to the source file do not corrupt the cache entry because the original content is now only reachable via the pointer.

  3. .gitignore patching. DVC automatically appends imagery.tif to .gitignore so Git never sees the raw binary. Only imagery.tif.dvc is staged and committed.

  4. Remote push. dvc push transfers cache entries to the remote defined in .dvc/config. The remote is environment-specific and is not embedded in the pointer file — a staging remote and a production remote can target the same .dvc files without modification:

    [core]
    remote = prod-s3
['remote "prod-s3"']
    url = s3://your-bucket/dvc-cache

  5. Pull and integrity verification. dvc pull downloads the binary from the remote using the MD5 from the pointer, then re-verifies the digest before placing the file. A mismatch raises a DvcException and aborts the checkout, preventing silent corruption.

Working Implementation

The following script automates DVC pointer creation for a batch of raster tiles, then verifies each pointer’s checksum before pushing to S3. It integrates naturally with the delta tracking algorithms used for vector change sets in the same pipeline.

#!/usr/bin/env python3
"""
batch_dvc_add.py

Add a directory of GeoTIFF raster tiles to DVC version control,
verify pointer integrity, and push to the configured S3 remote.

Requirements: dvc[s3]>=3.0, boto3
"""

import hashlib
import json
import subprocess
import sys
from pathlib import Path


RASTER_DIR = Path("data/rasters/sentinel2")
DVC_CACHE_DIR = Path(".dvc/cache")
EXPECTED_HASH_ALGO = "md5"


def md5_of_file(path: Path, chunk_size: int = 1 << 20) -> str:
    """Compute MD5 digest of a file in streaming chunks (memory-safe for large rasters)."""
    h = hashlib.md5()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(chunk_size), b""):
            h.update(chunk)
    return h.hexdigest()


def parse_dvc_pointer(dvc_file: Path) -> dict:
    """Read and return the first output entry from a .dvc YAML pointer."""
    import yaml  # PyYAML; installed with dvc
    data = yaml.safe_load(dvc_file.read_text())
    outs = data.get("outs", [])
    if not outs:
        raise ValueError(f"No outputs in {dvc_file}")
    return outs[0]


def verify_pointer_against_source(tif_path: Path, dvc_path: Path) -> bool:
    """
    Re-hash the source raster and compare against the MD5 stored in the
    .dvc pointer. Returns True if they match, raises on mismatch.
    """
    pointer = parse_dvc_pointer(dvc_path)
    stored_hash = pointer.get(EXPECTED_HASH_ALGO)
    if not stored_hash:
        raise KeyError(f"Pointer {dvc_path} has no '{EXPECTED_HASH_ALGO}' key")

    actual_hash = md5_of_file(tif_path)
    if actual_hash != stored_hash:
        raise RuntimeError(
            f"Hash mismatch for {tif_path.name}: "
            f"pointer={stored_hash}, actual={actual_hash}"
        )
    return True


def run(cmd: list[str], check: bool = True) -> subprocess.CompletedProcess:
    """Run a subprocess command and stream stdout/stderr."""
    print(f"  $ {' '.join(cmd)}")
    return subprocess.run(cmd, check=check, capture_output=False)


def main() -> int:
    tif_files = sorted(RASTER_DIR.glob("*.tif"))
    if not tif_files:
        print(f"No .tif files found in {RASTER_DIR}", file=sys.stderr)
        return 1

    print(f"Found {len(tif_files)} raster tile(s) — adding to DVC...")
    for tif in tif_files:
        # Stage with DVC; this writes <name>.tif.dvc and updates .gitignore
        run(["dvc", "add", str(tif)])

    print("\nVerifying pointer integrity...")
    errors = []
    for tif in tif_files:
        dvc_path = tif.with_suffix(".tif.dvc")
        try:
            verify_pointer_against_source(tif, dvc_path)
            print(f"  OK  {tif.name}")
        except Exception as exc:
            print(f"  FAIL {tif.name}: {exc}", file=sys.stderr)
            errors.append(tif.name)

    if errors:
        print(f"\n{len(errors)} integrity error(s). Aborting push.", file=sys.stderr)
        return 2

    print("\nPushing verified cache entries to remote...")
    run(["dvc", "push"])

    print("\nStaging .dvc pointer files for Git commit...")
    dvc_files = [str(tif.with_suffix(".tif.dvc")) for tif in tif_files]
    run(["git", "add"] + dvc_files + [".gitignore"])

    print("\nAll tiles tracked. Commit the staged .dvc files to complete versioning.")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Validation & Output Verification

After running the batch add, confirm pointer correctness and remote availability with these checks:

Pointer format validation:

# Confirm every .tif has a corresponding .dvc pointer
find data/rasters/sentinel2 -name "*.tif" | while read f; do
  dvc_file="${f}.dvc"
  [ -f "$dvc_file" ] || echo "MISSING POINTER: $dvc_file"
done

# Print the stored MD5 for spot-check comparison
python3 -c "
import yaml, sys
for p in sys.argv[1:]:
    d = yaml.safe_load(open(p))
    print(p, '-->', d['outs'][0].get('md5','no-md5'))
" data/rasters/sentinel2/*.tif.dvc

Remote availability check:

# List files DVC knows about on the remote without downloading them
dvc ls --dvc-only --recursive .

# Verify a specific tile is present in the remote cache
dvc status --cloud data/rasters/sentinel2/tile_001.tif.dvc

GeoGig pointer verification (LFS-backed rasters):

# Confirm LFS pointer was written (not the raw binary)
git show HEAD:data/rasters/sentinel2/tile_001.tif | head -3
# Expected output:
# version https://git-lfs.github.com/spec/v1
# oid sha256:<64-char hex>
# size <bytes>

# Verify LFS server holds the object
git lfs ls-files --long | grep tile_001.tif

Row-count / band assertion for raster integrity:

import rasterio
with rasterio.open("data/rasters/sentinel2/tile_001.tif") as src:
    assert src.count == 4, f"Expected 4 bands, got {src.count}"
    assert src.crs.to_epsg() == 32632, f"Expected EPSG:32632, got {src.crs.to_epsg()}"
    print("Band count and CRS OK")

Failure Modes

  • Symptom: git checkout leaves the raster as a raw text pointer file (three lines of version https://…). Root cause: Git LFS client is not installed or git lfs install was never run in this clone. Fix: git lfs install && git lfs pull.

  • Symptom: dvc pull fails with ERROR: failed to pull data from the cloud. Root cause: Remote credentials are missing or expired; the remote URL in .dvc/config points to a non-existent bucket path. Fix: Re-export AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (or the equivalent for GCS/Azure), then run dvc remote list to confirm the remote URL is reachable.

  • Symptom: dvc push succeeds but dvc status --cloud shows files as not in cache. Root cause: The local DVC cache was manually deleted or the working directory raster was modified after dvc add without re-running dvc add. Fix: dvc add <file> again to regenerate the pointer with the current checksum, then dvc push.

  • Symptom: GeoGig spatial diff returns no geometry changes despite obvious polygon edits. Root cause: The raster exceeded the LFS threshold so GeoGig routed it through LFS as an opaque blob; geometry-aware diffing only operates on natively parsed vector objects. Fix: Convert the raster-derived boundaries to a vector layer (geogig shp import) and diff the vector representation alongside the LFS-backed raster reference.