Migrating a Shapefile Repo from Git-LFS to DVC

A concrete migration runbook for teams outgrowing a Git-LFS store and moving Shapefile data onto DVC and cloud object storage — part of DVC vs GeoGit vs Git-LFS for Vector Datasets.

Concept & Context

Teams usually reach for Git-LFS first because it is one command away from a working setup. The friction appears later: LFS storage is billed and quota-limited on most hosted Git platforms, its remote is coupled to that host, and it offers none of the pipeline or cloud-backend flexibility that a growing spatial workflow needs. Migrating to DVC keeps the small pointers-in-Git model you already understand while moving the blobs to storage you control — an S3, GCS, or SSH remote with your own lifecycle rules.

The migration itself is mechanical and reversible if you stage it carefully. The core moves are: make sure every LFS pointer is materialized into a real file, untrack the patterns from LFS, hand the files to DVC, and push the blobs to a remote. The tradeoffs between the two tools, and why a team lands on DVC, are laid out in the parent guide on choosing between DVC and Git-LFS; the deeper mechanics of DVC’s cache and partial fetch are in large file handling in DVC for GIS.

Core Migration Pipeline

Run the migration on a dedicated branch so you can validate before anyone else pulls. The pipeline is six ordered steps; do not reorder them, because untracking from LFS before the files are materialized will leave you with pointer stubs and no data.

  1. Audit — enumerate every LFS-tracked path and its size, and snapshot feature counts per layer as a ground-truth baseline.
  2. Materialize — git lfs pull so every pointer becomes the real Shapefile on disk.
  3. Untrack — remove patterns from .gitattributes and run git lfs untrack, returning the files to ordinary working-tree status.
  4. Add to DVC — dvc add each Shapefile set (as a directory so sidecars stay together), producing .dvc pointers.
  5. Remote & push — configure a default DVC remote and dvc push the blobs to object storage.
  6. Verify — compare hashes and feature counts against the baseline, confirm a clean dvc status, then commit.

Working Implementation

The script below performs the whole migration for a repository whose Shapefiles live under data/. It is deliberately conservative: it materializes first, records a baseline, and refuses to proceed if any layer fails to open. Run it on a fresh branch.

#!/usr/bin/env bash
set -euo pipefail

DATA_DIR="data"
REMOTE_NAME="storage"
REMOTE_URL="s3://gis-artifacts/parcels-migration"

# --- Step 1: Audit what LFS currently tracks -------------------------------
echo "== LFS-tracked files =="
git lfs ls-files --long | tee /tmp/lfs_audit.txt

# Record a feature-count baseline per Shapefile before touching anything.
python - "$DATA_DIR" <<'PY'
import sys, pathlib, json
import geopandas as gpd
data = pathlib.Path(sys.argv[1])
baseline = {}
for shp in sorted(data.rglob("*.shp")):
    gdf = gpd.read_file(shp)
    baseline[str(shp)] = {"features": len(gdf), "crs": str(gdf.crs)}
pathlib.Path("/tmp/baseline.json").write_text(json.dumps(baseline, indent=2))
print(f"Baselined {len(baseline)} layer(s)")
PY

# --- Step 2: Materialize every pointer into a real file --------------------
git lfs pull

# --- Step 3: Untrack the patterns from Git-LFS -----------------------------
# Untrack each pattern that was in .gitattributes (adjust to your patterns).
git lfs untrack "*.shp" "*.shx" "*.dbf" "*.prj" "*.cpg" || true
git add .gitattributes

# --- Step 4: Hand each Shapefile set to DVC as a directory -----------------
# Tracking the parent folder keeps every sidecar (.shp/.shx/.dbf/.prj) together.
for layer_dir in $(find "$DATA_DIR" -type d -mindepth 1 -maxdepth 1); do
    dvc add "$layer_dir"
    git add "${layer_dir}.dvc" "$(dirname "$layer_dir")/.gitignore"
done

# --- Step 5: Configure the remote and push blobs off the Git host ----------
dvc remote add -d "$REMOTE_NAME" "$REMOTE_URL"
git add .dvc/config
dvc push

# --- Step 6: Verify feature counts survived the round-trip -----------------
python - "$DATA_DIR" <<'PY'
import sys, pathlib, json
import geopandas as gpd
baseline = json.loads(pathlib.Path("/tmp/baseline.json").read_text())
data = pathlib.Path(sys.argv[1])
for shp_str, expected in baseline.items():
    gdf = gpd.read_file(shp_str)
    assert len(gdf) == expected["features"], (
        f"{shp_str}: {len(gdf)} != {expected['features']} features")
    assert str(gdf.crs) == expected["crs"], f"{shp_str}: CRS drift"
print("All layers verified against baseline")
PY

git commit -m "migrate: move Shapefile data from Git-LFS to DVC"
echo "Migration complete. Run 'dvc status --cloud' to confirm remote sync."

The script keeps the storage-tool migration and any format conversion strictly separate — it never rewrites a Shapefile into another format. If you also want to leave the Shapefile format behind, do that as a distinct follow-up commit using the convert Shapefiles to GeoParquet workflow, so a verification failure is always attributable to one change.

On rewriting history

New commits now use DVC while old commits keep their LFS pointers, and the two coexist without conflict. You only need to rewrite history if you must reclaim the storage the old LFS objects occupy or completely sever the LFS dependency. That is a git filter-repo operation that rewrites every commit hash and forces all collaborators to re-clone — schedule it as a separate, announced step, never as part of this migration branch.

Validation & Output Verification

Beyond the in-script assertions, run these checks before merging the migration branch and before anyone decommissions the LFS store.

# 1. No file is still LFS-tracked
test -z "$(git lfs ls-files)" && echo "LFS clean" || echo "LFS still tracks files!"

# 2. DVC pointers resolve and the cache matches
dvc status            # expect: "Data and pipelines are up to date."
dvc status --cloud    # expect: nothing to push

# 3. A fresh checkout in a temp dir reproduces the data
git clone . /tmp/migration-check
cd /tmp/migration-check
dvc pull
python -c "import geopandas as gpd; print(len(gpd.read_file('data/parcels/parcels.shp')))"

The clean-clone check is the one that matters most: it proves a new teammate or a CI runner can reconstruct the working tree from the committed .dvc pointers plus the remote alone, with no dependency on your local cache.

Failure Modes

  • Symptom: After untracking, files show up as tiny text stubs. Root cause: git lfs untrack ran before git lfs pull, so the pointers were never materialized. Fix: Restore the branch, run git lfs pull first, then untrack.

  • Symptom: A checkout opens the layer but every attribute column is gone. Root cause: Only the .shp was added to DVC and the .dbf sidecar was left behind. Fix: Track the whole layer directory with dvc add, not the individual .shp, so all sidecars travel together.

  • Symptom: dvc push fails with an access-denied error. Root cause: The runner or your shell lacks credentials for the object store, or the remote region is wrong. Fix: Export storage credentials and set dvc remote modify storage region <region> to match the bucket.

  • Symptom: Repository size on the Git host barely drops after migration. Root cause: Old LFS objects still occupy the LFS store; new commits use DVC but history still references LFS blobs. Fix: This is expected — reclaim space only with a deliberate, coordinated git filter-repo history rewrite.

Back to DVC vs GeoGit vs Git-LFS for Vector Datasets