Migrating a Shapefile Repo from Git-LFS to DVC

A concrete migration runbook for teams outgrowing a Git-LFS store and moving Shapefile data onto DVC and cloud object storage — part of DVC vs GeoGit vs Git-LFS for Vector Datasets.

Concept & Context

Teams usually reach for Git-LFS first because it is one command away from a working setup. The friction appears later: LFS storage is billed and quota-limited on most hosted Git platforms, its remote is coupled to that host, and it offers none of the pipeline or cloud-backend flexibility that a growing spatial workflow needs. Migrating to DVC keeps the small pointers-in-Git model you already understand while moving the blobs to storage you control — an S3, GCS, or SSH remote with your own lifecycle rules.

The migration itself is mechanical and reversible if you stage it carefully. The core moves are: make sure every LFS pointer is materialized into a real file, untrack the patterns from LFS, hand the files to DVC, and push the blobs to a remote. The tradeoffs between the two tools, and why a team lands on DVC, are laid out in the parent guide on choosing between DVC and Git-LFS; the deeper mechanics of DVC’s cache and partial fetch are in large file handling in DVC for GIS.

Core Migration Pipeline

Run the migration on a dedicated branch so you can validate before anyone else pulls. The pipeline is six ordered steps; do not reorder them, because untracking from LFS before the files are materialized will leave you with pointer stubs and no data.

  1. Audit — enumerate every LFS-tracked path and its size, and snapshot feature counts per layer as a ground-truth baseline.
  2. Materialize — git lfs pull so every pointer becomes the real Shapefile on disk.
  3. Untrack — remove patterns from .gitattributes and run git lfs untrack, returning the files to ordinary working-tree status.
  4. Add to DVC — dvc add each Shapefile set (as a directory so sidecars stay together), producing .dvc pointers.
  5. Remote & push — configure a default DVC remote and dvc push the blobs to object storage.
  6. Verify — compare hashes and feature counts against the baseline, confirm a clean dvc status, then commit.
Migrating without a day where nothing works Six steps: inventory what LFS currently tracks, stand up the DVC remote alongside it, track the same artifacts with DVC while LFS still works, verify hashes match on both sides, cut the team over to DVC, and only then stop tracking with LFS. 1 Inventory what LFS tracks patterns, sizes, and who pulls them 2 Stand up the DVC remote alongside LFS, not instead of it 3 Track the same artifacts with DVC both systems hold the bytes for a while 4 Verify hashes on both sides same content, two content addresses 5 Cut the team over one announcement, one documented command 6 Untrack from LFS last, and only after a quiet week The overlap in steps three to five is what turns a migration into a reversible change.

Working Implementation

The script below performs the whole migration for a repository whose Shapefiles live under data/. It is deliberately conservative: it materializes first, records a baseline, and refuses to proceed if any layer fails to open. Run it on a fresh branch.

#!/usr/bin/env bash
set -euo pipefail

DATA_DIR="data"
REMOTE_NAME="storage"
REMOTE_URL="s3://gis-artifacts/parcels-migration"

# --- Step 1: Audit what LFS currently tracks -------------------------------
echo "== LFS-tracked files =="
git lfs ls-files --long | tee /tmp/lfs_audit.txt

# Record a feature-count baseline per Shapefile before touching anything.
python - "$DATA_DIR" <<'PY'
import sys, pathlib, json
import geopandas as gpd
data = pathlib.Path(sys.argv[1])
baseline = {}
for shp in sorted(data.rglob("*.shp")):
    gdf = gpd.read_file(shp)
    baseline[str(shp)] = {"features": len(gdf), "crs": str(gdf.crs)}
pathlib.Path("/tmp/baseline.json").write_text(json.dumps(baseline, indent=2))
print(f"Baselined {len(baseline)} layer(s)")
PY

# --- Step 2: Materialize every pointer into a real file --------------------
git lfs pull

# --- Step 3: Untrack the patterns from Git-LFS -----------------------------
# Untrack each pattern that was in .gitattributes (adjust to your patterns).
git lfs untrack "*.shp" "*.shx" "*.dbf" "*.prj" "*.cpg" || true
git add .gitattributes

# --- Step 4: Hand each Shapefile set to DVC as a directory -----------------
# Tracking the parent folder keeps every sidecar (.shp/.shx/.dbf/.prj) together.
for layer_dir in $(find "$DATA_DIR" -type d -mindepth 1 -maxdepth 1); do
    dvc add "$layer_dir"
    git add "${layer_dir}.dvc" "$(dirname "$layer_dir")/.gitignore"
done

# --- Step 5: Configure the remote and push blobs off the Git host ----------
dvc remote add -d "$REMOTE_NAME" "$REMOTE_URL"
git add .dvc/config
dvc push

# --- Step 6: Verify feature counts survived the round-trip -----------------
python - "$DATA_DIR" <<'PY'
import sys, pathlib, json
import geopandas as gpd
baseline = json.loads(pathlib.Path("/tmp/baseline.json").read_text())
data = pathlib.Path(sys.argv[1])
for shp_str, expected in baseline.items():
    gdf = gpd.read_file(shp_str)
    assert len(gdf) == expected["features"], (
        f"{shp_str}: {len(gdf)} != {expected['features']} features")
    assert str(gdf.crs) == expected["crs"], f"{shp_str}: CRS drift"
print("All layers verified against baseline")
PY

git commit -m "migrate: move Shapefile data from Git-LFS to DVC"
echo "Migration complete. Run 'dvc status --cloud' to confirm remote sync."

The script keeps the storage-tool migration and any format conversion strictly separate — it never rewrites a Shapefile into another format. If you also want to leave the Shapefile format behind, do that as a distinct follow-up commit using the convert Shapefiles to GeoParquet workflow, so a verification failure is always attributable to one change.

On rewriting history

New commits now use DVC while old commits keep their LFS pointers, and the two coexist without conflict. You only need to rewrite history if you must reclaim the storage the old LFS objects occupy or completely sever the LFS dependency. That is a git filter-repo operation that rewrites every commit hash and forces all collaborators to re-clone — schedule it as a separate, announced step, never as part of this migration branch.

A cutover measured in weeks, not minutes A timeline of the migration: both systems track the artifacts during an overlap week, the team switches its daily commands, LFS tracking is removed once nobody reports a problem, and the LFS objects are retired after the agreed retention window. Dual tracking both systems hold every artifact week 1 Team cutover documented commands change week 2 LFS untracked history untouched week 3 LFS objects retired after the retention window week 12 Nothing is deleted until the last marker, which is what makes every earlier step safe to undo. Rewrite the history, or leave it alone? Two panels weighing a history rewrite against leaving LFS pointers in past commits. Rewriting removes the old dependency completely at the cost of invalidating every clone, fork, and commit reference; leaving history alone keeps every reference valid at the cost of needing LFS to check out old revisions. REWRITE HISTORY The LFS dependency disappears completely Every clone, fork and open branch must be recreated Every commit hash in a ticket or a paper breaks One coordinated event, with a rollback plan LEAVE HISTORY ALONE Every existing reference keeps working Old revisions still need LFS to check out The LFS server must stay alive, read-only No coordination needed at all Leave it alone unless the LFS server is going away — the rewrite costs more than it saves. A read-only LFS server is cheap; a broken commit reference in a published paper is not.

Validation & Output Verification

Beyond the in-script assertions, run these checks before merging the migration branch and before anyone decommissions the LFS store.

# 1. No file is still LFS-tracked
test -z "$(git lfs ls-files)" && echo "LFS clean" || echo "LFS still tracks files!"

# 2. DVC pointers resolve and the cache matches
dvc status            # expect: "Data and pipelines are up to date."
dvc status --cloud    # expect: nothing to push

# 3. A fresh checkout in a temp dir reproduces the data
git clone . /tmp/migration-check
cd /tmp/migration-check
dvc pull
python -c "import geopandas as gpd; print(len(gpd.read_file('data/parcels/parcels.shp')))"

The clean-clone check is the one that matters most: it proves a new teammate or a CI runner can reconstruct the working tree from the committed .dvc pointers plus the remote alone, with no dependency on your local cache.

Failure Modes

  • Symptom: After untracking, files show up as tiny text stubs. Root cause: git lfs untrack ran before git lfs pull, so the pointers were never materialized. Fix: Restore the branch, run git lfs pull first, then untrack.

  • Symptom: A checkout opens the layer but every attribute column is gone. Root cause: Only the .shp was added to DVC and the .dbf sidecar was left behind. Fix: Track the whole layer directory with dvc add, not the individual .shp, so all sidecars travel together.

  • Symptom: dvc push fails with an access-denied error. Root cause: The runner or your shell lacks credentials for the object store, or the remote region is wrong. Fix: Export storage credentials and set dvc remote modify storage region <region> to match the bucket.

  • Symptom: Repository size on the Git host barely drops after migration. Root cause: Old LFS objects still occupy the LFS store; new commits use DVC but history still references LFS blobs. Fix: This is expected — reclaim space only with a deliberate, coordinated git filter-repo history rewrite.

Back to DVC vs GeoGit vs Git-LFS for Vector Datasets