Choosing a Remote Backend for a Distributed GIS Team

Pointer-based versioning decouples the bytes from the repository, which makes where those bytes live a decision in its own right — and one that shows up every time somebody clones. This page is a focused companion to DVC vs GeoGit vs Git LFS for vector datasets.

Concept & Context

The tool comparison usually ends with a choice of pointer system, and the backend gets picked afterwards by whoever sets it up. That inverts the actual impact. The pointer system determines the workflow; the backend determines whether a colleague in another country waits four minutes or forty for the data that workflow describes.

Three properties dominate for spatial repositories. Concurrency matters more than raw throughput, because a spatial dataset is a lumpy mixture of a few very large artifacts and many small ones, and a backend that parallelises well moves the small ones while a large one streams. Egress cost matters more than storage cost for a distributed team, since the same release is fetched by everyone. And access control granularity decides whether a sensitivity tiering can be enforced at the storage layer or only by convention.

Performance is the last thing to evaluate, not the first. A backend that cannot express per-prefix access policy, or cannot restore an object deleted last Tuesday, is disqualified before its speed is interesting.

Core Algorithmic Pipeline

  1. Characterise the access pattern: office locations, typical pull size, and how often the same bytes are fetched by more than one person.
  2. Screen on governance: per-prefix policy, versioning, object lock, audit logging, restore window.
  3. Measure the survivors on a real subset of your own data, cold cache, several concurrency levels.
  4. Price transfer and requests at the measured volume, not the storage footprint.
  5. Add a read-through cache per office if the same releases are fetched repeatedly, and re-measure.
The screening that happens before any benchmark A grid of four governance properties — per-prefix policy, object versioning, access logging and restore window — against what each one enables and what its absence disqualifies. Enables Absence disqualifies Per-prefix policy sensitivity tiering at the storage layer any layer above internal Object versioning recovering an overwritten release anything with a published release Access logging attributing every read restricted and controlled tiers Restore window undoing an accidental delete nothing you would rather not lose A backend failing any of the first three is out, whatever it does on a throughput chart.

Working Implementation

"""Measure candidate remotes on your own repository, not on a synthetic file."""
from __future__ import annotations

import json
import statistics
import subprocess
import time
from dataclasses import dataclass


@dataclass
class Result:
    backend: str
    jobs: int
    seconds: float
    bytes_moved: int

    @property
    def mib_per_s(self) -> float:
        return self.bytes_moved / self.seconds / (1024 ** 2)


def clear_local_cache() -> None:
    """A warm cache measures your disk, which is not the question."""
    subprocess.run(["rm", "-rf", ".dvc/cache"], check=True)


def pull(remote: str, jobs: int, target: str = "data/") -> Result:
    clear_local_cache()
    start = time.monotonic()
    subprocess.run(["dvc", "pull", target, "--remote", remote, "--jobs", str(jobs)],
                   check=True, capture_output=True)
    elapsed = time.monotonic() - start

    size = int(subprocess.check_output(
        ["du", "-sb", "--", "data"], text=True).split()[0])
    return Result(remote, jobs, elapsed, size)


def sweep(remotes: list[str], job_counts=(1, 4, 16, 32), repeats: int = 3) -> list[dict]:
    rows = []
    for remote in remotes:
        for jobs in job_counts:
            runs = [pull(remote, jobs) for _ in range(repeats)]
            rows.append({
                "backend": remote,
                "jobs": jobs,
                "median_s": round(statistics.median(r.seconds for r in runs), 1),
                "spread_s": round(max(r.seconds for r in runs)
                                  - min(r.seconds for r in runs), 1),
                "median_mib_s": round(statistics.median(r.mib_per_s for r in runs), 1),
            })
            print(f"{remote:<18} jobs={jobs:<3} "
                  f"{rows[-1]['median_s']:>7.1f}s  "
                  f"{rows[-1]['median_mib_s']:>7.1f} MiB/s")
    return rows


def monthly_transfer_cost(rows: list[dict], repo_gb: float, pulls_per_month: int,
                          egress_per_gb: dict[str, float]) -> dict[str, float]:
    """The bill nobody models until it arrives."""
    return {
        backend: round(repo_gb * pulls_per_month * price, 2)
        for backend, price in egress_per_gb.items()
    }

A representative sweep on a 6 GB repository, measured from a second region:

backend            jobs=1     jobs=4    jobs=16    jobs=32
object-store        512.4s     161.2s     71.8s      68.9s
shared-mount        184.6s     176.3s    174.1s     173.4s
self-hosted-minio   402.7s     122.5s     58.3s      55.1s

The shape matters more than the absolute numbers. The mount is fastest at one job and does not improve, because it is a single stream over a link that parallelism cannot widen. The object stores start slower and overtake decisively by four jobs — which is why a team that concluded “the mount is faster” from a single-threaded test reached the wrong answer.

Governance screening happens before any of this, and it is a checklist rather than a measurement:

GOVERNANCE = {
    "per_prefix_policy":   "can read/write be scoped per sensitivity tier?",
    "object_versioning":   "can an overwritten object be recovered?",
    "object_lock":         "can a release be made immutable for a retention period?",
    "access_logging":      "is every read attributable to a principal?",
    "restore_window":      "how long after a delete can an object be restored?",
    "regional_placement":  "can data be pinned to a jurisdiction?",
}

A backend failing object_versioning or access_logging cannot support the tiering described in security boundaries in spatial repositories, whatever it does on a throughput chart.

The measurement that changes the answer Horizontal bars of cold checkout time for a 6.2 GB repository from a second region, at one job and at sixteen, for a shared mount and an object store — showing that the mount wins single-threaded and loses decisively with concurrency. COLD CHECKOUT, 6.2 GB, FROM A SECOND REGION Mount, 1 job 185 s the number a single-threaded test reports Mount, 16 jobs 174 s one stream; parallelism cannot widen it Object store, 1 job 512 s Object store, 16 jobs 72 s A team that benchmarks at one job concludes the mount is faster, and lives with it for years.

Validation & Output Verification

# Governance first — these must be answered before a benchmark is worth running
aws s3api get-bucket-versioning --bucket gis-artifacts
aws s3api get-object-lock-configuration --bucket gis-artifacts
aws s3api get-bucket-logging --bucket gis-artifacts

# Then measure, cold, several times, at more than one concurrency
python scripts/bench_remote.py --remotes object-store shared-mount \
  --jobs 1 4 16 32 --repeats 3 | tee bench.txt
# The choice must survive its own numbers: check the spread, not just the median
import json
rows = json.load(open("bench.json"))
for row in rows:
    if row["spread_s"] > 0.3 * row["median_s"]:
        print(f"unstable: {row['backend']} at {row['jobs']} jobs — "
              f"spread {row['spread_s']}s on a median of {row['median_s']}s")

# And confirm a scoped credential really is scoped
import subprocess
r = subprocess.run(["dvc", "push", "--remote", "restricted-tier"],
                   capture_output=True, text=True)
assert r.returncode != 0, "a read-only credential was able to write"
print("tier credentials are scoped as configured")

Failure Modes

  • A single-threaded benchmark picks the wrong backend — symptom: the chosen remote is slow in daily use. Root cause: measuring at one job, where a mount looks best. Fix: sweep concurrency, since real pulls run parallel.

  • The transfer bill surprises everyone — symptom: egress dominating the cloud spend. Root cause: pricing modelled on storage. Fix: model transfer at the measured pull volume; co-locate CI with the store; add per-office read-through caches.

  • A release is overwritten — symptom: a tag now names different bytes. Root cause: no object versioning or lock, and a write credential shared with the promotion job. Fix: enable versioning and lock; separate read and write credentials per stage, as automating DVC push on merge does.

  • Remote workers time out — symptom: pulls fail for one office. Root cause: a shared mount over a VPN. Fix: move to object storage with a local cache; the mount’s advantage disappears the moment the network is not local.

What a per-office read-through cache changes A sequence in which the first colleague to fetch a release pulls it from the object store into the office cache, and the next three fetch the same release from the cache over the local network rather than across the internet. First fetch Office cache Object store Next three release v2026.08? miss — fetch and retain the only wide-area transfer same release served on the local network It also removes three quarters of the egress bill for that release, which is usually the larger saving.

Back to DVC vs GeoGit vs Git LFS for Vector Datasets