Recovering from a Broken Pointer After a Force Push

The rewrite succeeded, the branch looks clean, and dvc pull reports that half a terabyte of imagery cannot be found — because the pointers naming it are no longer in any reachable commit. This page is a focused companion to pointer synchronization for raster datasets.

Concept & Context

Pointer-based versioning splits a dataset into two halves that are kept in step by reference: Git holds small text files naming content hashes, and an object store holds the bytes those hashes address. Rewriting history changes the first half only. The blobs are untouched — but the record of which blob belonged to which version is what was rewritten, and without it a hash in the object store is an anonymous object nobody can attribute.

The saving grace is content addressing. A blob is identified by what it contains, so any surviving copy of the pointer — in a reflog, in a colleague’s un-fetched clone, in a CI cache, in a release manifest — is sufficient to reattach it. Recovery is therefore a search for a pointer, not for the data.

The clock matters. Object stores are pruned on a schedule, and a prune run after a rewrite will conclude that the orphaned blobs are unreferenced and delete them. The first action in a recovery is not to investigate; it is to stop the prune.

The recovery window, and what closes it A timeline from the force push to the loss of recoverability: the rewrite orphans the pointers, the blobs remain untouched in the object store, and a scheduled prune is what actually destroys them — which makes stopping the prune the first response. Force push pointers orphaned, blobs untouched T dvc pull fails the first visible symptom T+h Prune suspended the window stays open T+h Scheduled prune would delete the orphans T+24h Everything else in a recovery can wait; the third marker cannot.

Core Algorithmic Pipeline

  1. Suspend garbage collection on the object store and on any DVC cache prune job, before anything else.
  2. Enumerate the discarded commits from the reflog on any machine that had the old history, including CI runners.
  3. Extract every pointer those commits contained, and the hashes they name.
  4. Locate each hash in the remote, in a local cache, or in a colleague’s cache.
  5. Reattach by committing a pointer that names the recovered hash, then verify the artifact against it.
Finding a pointer that names the lost bytes A sequence searching for a surviving pointer: the reflog offers discarded commits, each commit is read for its pointer files, and the hashes those name are probed against the object store — so recovery is a search for a pointer rather than for the data. Operator Reflog Discarded commit Object store which commits did the rewrite discard? 17 unreachable commits git show <sha>:path.dvc md5 and size is this hash still present? content addressing — any copy will do The bytes were never lost. What was lost is the record of which bytes belonged to which version.

Working Implementation

#!/usr/bin/env bash
# recover_pointers.sh — reattach artifacts orphaned by a history rewrite.
set -euo pipefail

echo "==> 1. Stop anything that prunes. Do this first, investigate second."
# (Disable the scheduled `dvc gc` job and any object-store lifecycle rule now.)

echo "==> 2. Commits the rewrite discarded, from every reflog we can reach"
git reflog --date=iso --all \
  | awk '{print $1}' | sort -u > /tmp/reflog-commits.txt
git fsck --lost-found --no-progress 2>/dev/null \
  | awk '/dangling commit/ {print $3}' >> /tmp/reflog-commits.txt
sort -u -o /tmp/reflog-commits.txt /tmp/reflog-commits.txt
wc -l < /tmp/reflog-commits.txt
#!/usr/bin/env python3
"""Enumerate orphaned artifact hashes and report where each one can be found."""
from __future__ import annotations

import json
import subprocess
from pathlib import Path

import yaml


def pointers_in(commit: str) -> dict[str, dict]:
    """Every .dvc pointer in a commit, mapped path -> {md5, size}."""
    try:
        listing = subprocess.check_output(
            ["git", "ls-tree", "-r", "--name-only", commit],
            text=True, stderr=subprocess.DEVNULL,
        ).split()
    except subprocess.CalledProcessError:
        return {}

    found = {}
    for path in (p for p in listing if p.endswith(".dvc")):
        blob = subprocess.check_output(["git", "show", f"{commit}:{path}"], text=True)
        for out in yaml.safe_load(blob).get("outs", []):
            if "md5" in out:
                found[path] = {"md5": out["md5"], "size": out.get("size")}
    return found


def reachable_hashes() -> set[str]:
    """Hashes referenced by history that still exists."""
    live = set()
    commits = subprocess.check_output(["git", "rev-list", "--all"], text=True).split()
    for commit in commits:
        for info in pointers_in(commit).values():
            live.add(info["md5"])
    return live


def locate(md5: str, cache_dir: Path) -> str:
    """Where a given hash can be found, if anywhere."""
    local = cache_dir / md5[:2] / md5[2:]
    if local.exists():
        return f"local cache {local}"

    probe = subprocess.run(
        ["dvc", "get", ".", "--show-url", md5], capture_output=True, text=True
    )
    if probe.returncode == 0:
        return "remote"
    return "MISSING"


def main() -> int:
    cache_dir = Path(".dvc/cache/files/md5")
    live = reachable_hashes()

    orphaned: dict[str, dict] = {}
    for commit in Path("/tmp/reflog-commits.txt").read_text().split():
        for path, info in pointers_in(commit).items():
            if info["md5"] not in live:
                orphaned.setdefault(info["md5"], {"paths": set(), **info})
                orphaned[info["md5"]]["paths"].add(path)

    report = []
    for md5, info in sorted(orphaned.items()):
        where = locate(md5, cache_dir)
        report.append({"md5": md5, "paths": sorted(info["paths"]),
                       "size": info["size"], "found_in": where})
        marker = "!" if where == "MISSING" else " "
        print(f"{marker} {md5[:12]}  {where:<28} {sorted(info['paths'])[0]}")

    Path("orphaned-artifacts.json").write_text(json.dumps(report, indent=2))
    missing = sum(1 for r in report if r["found_in"] == "MISSING")
    print(f"\n{len(report)} orphaned artifact(s); {missing} not found anywhere")
    return 1 if missing else 0


if __name__ == "__main__":
    raise SystemExit(main())

Reattaching restores the pointer rather than the bytes, which is the distinction that preserves reproducibility:

# Restore the pointer exactly as the discarded commit held it
git show "${LOST_COMMIT}:data/raw/orthomosaic_2025.tif.dvc" \
  > data/raw/orthomosaic_2025.tif.dvc

# Pull the artifact the restored pointer names, and verify it
dvc pull data/raw/orthomosaic_2025.tif.dvc
dvc status data/raw/orthomosaic_2025.tif.dvc     # expected: up to date

git add data/raw/orthomosaic_2025.tif.dvc
git commit -m "Restore pointer for orthomosaic_2025 orphaned by the 2026-08-04 rewrite"

Validation & Output Verification

# Every pointer in every reachable commit must resolve on the remote
git rev-list --all | while read -r sha; do
  git ls-tree -r --name-only "$sha" | grep '\.dvc$' | while read -r ptr; do
    md5=$(git show "$sha:$ptr" | awk '/md5:/ {print $2}')
    dvc get . --show-url "$md5" >/dev/null 2>&1 \
      || echo "UNRESOLVABLE $sha $ptr $md5"
  done
done | sort -u

# Release tags are the ones that must never break — check them explicitly
git tag -l 'v*' | while read -r tag; do
  git checkout -q "$tag" 2>/dev/null || continue
  dvc status --cloud >/dev/null 2>&1 \
    && echo "OK   $tag" || echo "BROKEN $tag"
done
git checkout -q main

Run the tag check on a schedule rather than only after an incident. It is a few minutes of object-store queries and it turns “a release is unreproducible” from something a user discovers into something a job reports.

Failure Modes

  • The blobs were pruned during the investigation — symptom: hashes that existed an hour ago are gone. Root cause: a scheduled prune ran against rewritten history. Fix: suspend pruning as the first action; restore from backup if it already ran.

  • The recovered artifact has a different hash — symptom: dvc status reports a modification after recovery. Root cause: the artifact was re-added from the working tree instead of the pointer being restored. Fix: restore the pointer from the discarded commit and pull the bytes it names.

  • Reflog is empty on every machine — symptom: nothing knows the old commits. Root cause: the rewrite happened long ago, or reflog expiry ran. Fix: recover hashes from release manifests, which is the reason to keep an out-of-band inventory of hashes per tag.

  • The rewrite is repeated a month later — symptom: the same incident. Root cause: force pushes are still permitted on the default branch. Fix: protect the branch, and add the pointer-resolution check to the pipeline so a rewrite that orphans anything fails before it is merged.

Restoring the pointer, versus re-adding the file Two panels contrasting the two recovery routes. Re-adding the working-tree file hashes whatever bytes are present now and produces a new address; restoring the pointer from the discarded commit names the original hash, so the historical version is genuinely recovered. RE-ADD THE FILE Hashes whatever is in the working tree now Produces a new address if anything differs at all The tag that referenced the original still cannot resolve Looks successful, and is not RESTORE THE POINTER Names the hash the discarded commit recorded Pulls and verifies the original bytes The historical version resolves again Fails loudly if the blob is genuinely gone Recover the reference, not the file — the file was never the thing that was lost. The distinction is invisible until someone tries to reproduce the release that referenced it.

Back to Pointer Synchronization for Raster Datasets