Benchmarking Checkout Speed Across Versioning Tools

A reproducible harness that times how long each tool takes to materialize a vector dataset into a working tree — part of DVC vs GeoGit vs Git-LFS for Vector Datasets.

Concept & Context

Vendor-neutral timing numbers are the only way to settle a versioning-tool debate on your own hardware, network, and dataset shape. The headline figures in the parent comparison of DVC, Git-LFS, and GeoGit are useful as a starting orientation, but checkout speed is dominated by variables that differ per team: link bandwidth to the remote, local disk speed, dataset compressibility, and whether the cache is warm. A harness that you can run in an afternoon replaces argument with measurement.

The subtlety that trips up most ad hoc benchmarks is cache state. git lfs pull and dvc pull both consult a local cache before touching the network, so a second run of the “same” command can be ten times faster than the first while measuring something entirely different. A credible harness controls this explicitly, timing a cold run against an emptied cache and a warm run against a populated one, and never blends the two. The same discipline underpins the operational guidance in large file handling in DVC for GIS.

Core Benchmark Pipeline

The harness runs the same five phases for every tool and dataset size, keeping inputs identical so the only variable is the tool under test.

1. Generate fixtures — synthesize GeoPackage layers at a small, medium, and large size (for example 50 MB, 500 MB, 2 GB) with a fixed random seed so every tool sees byte-identical data.
2. Set up each tool — commit the fixture under plain Git, Git-LFS, and DVC in isolated clones so their caches and remotes do not interfere.
3. Control cache state — before a cold run, purge the tool’s cache (dvc cache directory or the LFS cache); before a warm run, pre-populate it once and leave it.
4. Time the operation — wrap git checkout, git lfs pull, and dvc checkout/dvc pull in a monotonic timer, repeating each measurement enough times for a stable median.
5. Aggregate and report — collect every timing into a tidy table keyed by tool, size, and cache state, reporting medians and interquartile spread rather than means.

Working Implementation

The harness below is a single Python module that drives the tool commands through subprocess, controls cache state between runs, and emits a Markdown results table. It assumes the three repositories are already initialized (plain Git, LFS, DVC) and that a FIXTURES mapping points at each dataset size. Timing uses time.monotonic to avoid wall-clock adjustments.

#!/usr/bin/env python3
"""Benchmark checkout/pull speed across git, git-lfs, and dvc."""
import shutil
import statistics
import subprocess
import time
from dataclasses import dataclass
from pathlib import Path

# Map a size label to (repo_path, cache_path, materialize_command).
# Each repo is a separate clone so caches never collide.
FIXTURES = {
    "50MB":  Path("./bench/small"),
    "500MB": Path("./bench/medium"),
    "2GB":   Path("./bench/large"),
}
REPEATS = 7  # cold pulls are noisy; medians need enough samples


@dataclass
class ToolSpec:
    name: str
    cache_dir: str            # relative to repo; emptied for a cold run
    warm_cmd: list[str]       # command that materializes from a warm cache
    cold_cmd: list[str]       # command that materializes from cold (network)


TOOLS = {
    "git":     ToolSpec("git", ".git",       ["git", "checkout", "--", "data"],
                                              ["git", "checkout", "--", "data"]),
    "git-lfs": ToolSpec("git-lfs", ".git/lfs", ["git", "lfs", "checkout"],
                                                ["git", "lfs", "pull"]),
    "dvc":     ToolSpec("dvc", ".dvc/cache",  ["dvc", "checkout"],
                                              ["dvc", "pull"]),
}


def _run(cmd: list[str], cwd: Path) -> None:
    """Run a command, raising on failure so a broken tool never scores 0."""
    subprocess.run(cmd, cwd=cwd, check=True,
                   stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)


def _clear_cache(repo: Path, cache_dir: str) -> None:
    """Force a cold run by removing the tool's local cache."""
    target = repo / cache_dir
    if cache_dir == ".git":       # never delete the whole repo for plain git
        return
    if target.exists():
        shutil.rmtree(target, ignore_errors=True)


def time_once(repo: Path, cmd: list[str]) -> float:
    """Materialize once and return elapsed seconds (monotonic clock)."""
    # Remove materialized data first so the command has real work to do.
    data = repo / "data"
    if data.exists():
        shutil.rmtree(data, ignore_errors=True)
    start = time.monotonic()
    _run(cmd, cwd=repo)
    return time.monotonic() - start


def benchmark() -> list[dict]:
    rows: list[dict] = []
    for size, repo in FIXTURES.items():
        for tool_key, spec in TOOLS.items():
            for cache_state in ("cold", "warm"):
                samples: list[float] = []
                for i in range(REPEATS):
                    if cache_state == "cold":
                        _clear_cache(repo, spec.cache_dir)
                        cmd = spec.cold_cmd
                    else:
                        # First iteration warms the cache; time the rest.
                        cmd = spec.warm_cmd
                    elapsed = time_once(repo, cmd)
                    if not (cache_state == "warm" and i == 0):
                        samples.append(elapsed)
                median = statistics.median(samples)
                spread = (max(samples) - min(samples)) if samples else 0.0
                rows.append({
                    "size": size, "tool": tool_key, "cache": cache_state,
                    "median_s": round(median, 2), "spread_s": round(spread, 2),
                })
    return rows


def to_markdown(rows: list[dict]) -> str:
    header = "| Size | Tool | Cache | Median (s) | Spread (s) |\n"
    header += "|------|------|-------|-----------|-----------|\n"
    body = "\n".join(
        f"| {r['size']} | {r['tool']} | {r['cache']} | "
        f"{r['median_s']} | {r['spread_s']} |"
        for r in rows
    )
    return header + body


if __name__ == "__main__":
    results = benchmark()
    print(to_markdown(results))

The harness deliberately deletes the materialized data directory before each timed run so the command has genuine work to do — otherwise a no-op checkout would report near-zero and flatter every tool equally. It also skips the first warm iteration, which is the one that populates the cache, so warm numbers reflect steady state.

Example results

Running the harness against a synthetic parcel GeoPackage on a single CI runner with a 300 Mbps link to an S3 remote produced the table below. Plain Git has no separate cold/warm distinction for a local checkout, so its two rows are near-identical and network-independent.

The pattern is consistent: cold time scales with dataset size and link speed because it is network-bound, while warm time is an order of magnitude faster and disk-bound. DVC edges ahead of Git-LFS on cold pulls here mainly because of its parallel chunked transfer, but on a different link the gap can invert — which is exactly why you run this on your own infrastructure rather than trusting a published figure.

Validation & Output Verification

A benchmark is only trustworthy if each timed run actually produced the correct data. Add a post-run assertion that the materialized layer matches the fixture by content hash and feature count, so a silently-failed pull can never masquerade as a fast one.

import hashlib
import geopandas as gpd
from pathlib import Path


def verify_materialized(repo: Path, expected_sha256: str, expected_features: int):
    layer = repo / "data" / "parcels.gpkg"
    assert layer.exists(), f"materialize produced no file in {repo}"

    digest = hashlib.sha256(layer.read_bytes()).hexdigest()
    assert digest == expected_sha256, f"content hash mismatch in {repo}"

    gdf = gpd.read_file(layer)
    assert len(gdf) == expected_features, (
        f"{len(gdf)} != {expected_features} features in {repo}")
    print(f"{repo.name}: verified {expected_features} features, hash OK")

Wire verify_materialized into time_once right after _run returns and before the timer would be reused. Also confirm your cold runs are genuinely cold: log the cache directory size immediately before each cold command and assert it is empty, so a stale cache never inflates a “cold” result into a warm one.

Failure Modes

• Symptom: Cold and warm numbers are nearly identical for a tool. Root cause: The cache was never actually cleared, so every run read from a populated cache. Fix: Verify _clear_cache targets the right directory and assert the cache is empty before the cold command.

• Symptom: DVC or LFS times are suspiciously fast and constant across sizes. Root cause: The materialized data directory was not removed, so the command short-circuited as a no-op. Fix: Delete the working data before each timed run, as time_once does.

• Symptom: Wildly variable cold pulls with huge spread. Root cause: Contended network bandwidth or a throttling remote skews individual transfers. Fix: Increase REPEATS, report the median and interquartile range, and run during a quiet window.

• Symptom: Plain Git checkout throws or takes minutes. Root cause: The binary was committed directly to Git, so the object database is enormous and the checkout unpacks a huge blob. Fix: Expected behaviour — it is the baseline that pointer tools are meant to beat; keep the row for contrast.

Related

• DVC vs GeoGit vs Git-LFS for Vector Datasets — the decision guide these numbers inform
• Migrating a Shapefile Repo from Git-LFS to DVC — measure the before-and-after of a tool change
• Large File Handling in DVC for GIS — cache and remote tuning that moves these numbers
• Understanding Pointer Files in GeoGit vs DVC — why warm checkouts are so much cheaper than cold pulls

Back to DVC vs GeoGit vs Git-LFS for Vector Datasets