Enforcing File Size Budgets on Spatial Pull Requests

Somebody will eventually git add a 4 GB GeoTIFF instead of tracking it, and once it is in history removing it costs far more than the check that would have stopped it. This page is a focused companion to CI/CD validation pipelines for spatial repositories.

Concept & Context

The failure is mundane and permanent. A raster is copied into the working tree, added without a tracking step, committed and pushed. From then on every clone of the repository downloads it, forever, and the only remedy is a history rewrite that breaks every existing clone and every commit reference in a ticket or a paper.

A size budget catches this in seconds. What makes it slightly more interesting than find -size is that a spatial repository legitimately contains very large artifacts — they are simply behind pointers. So the check has to distinguish two things: a blob committed directly to Git, which is nearly always a mistake, and a tracked artifact that grew, which is often deliberate and occasionally a signal worth surfacing.

Budgets are per path because the answer differs by directory. data/raw/ may reasonably hold multi-gigabyte tracked artifacts. scripts/ should hold nothing above a megabyte. data/interim/ should perhaps hold nothing at all, since it is derived.

Core Algorithmic Pipeline

  1. Declare budgets per path prefix, with separate limits for Git blobs and for tracked artifacts.
  2. Enumerate what the branch adds relative to the merge base, not the whole tree.
  3. Classify each addition: a raw Git blob, a pointer file, or a pointer whose artifact grew.
  4. Compare against the budget for the most specific matching prefix.
  5. Fail with the numbers, and allow an override that records who granted it and why.
Deciding what a changed path is, before measuring it A decision ladder over each changed path: a pointer file is measured by the artifact it names, a path outside every budget rule is ignored, and anything else is measured as a raw Git blob against the blob budget for its directory. Is the path a pointer file? Measure the artifact the size the pointer records yes no Does any budget rule match the path? Measure the git blob against the blob budget yes no Not budgeted left alone deliberately Separating the two measurements is what lets a tracked eight-gigabyte raster pass while a one-megabyte script fails.

Working Implementation

#!/usr/bin/env python3
"""Enforce per-path size budgets on the artifacts a branch adds."""
from __future__ import annotations

import subprocess
import sys
from pathlib import Path

import yaml

MB = 1024 * 1024


def load_budgets(path: str = ".size-budgets.yml") -> list[dict]:
    """Most specific prefix wins, so sort longest-first once at load time."""
    raw = yaml.safe_load(Path(path).read_text())["budgets"]
    return sorted(raw, key=lambda b: len(b["path"]), reverse=True)


def budget_for(file_path: str, budgets: list[dict]) -> dict | None:
    for b in budgets:
        if file_path.startswith(b["path"]):
            return b
    return None


def added_files(base: str) -> list[tuple[str, int]]:
    """(path, size in bytes) for every file this branch adds or changes."""
    names = subprocess.check_output(
        ["git", "diff", "--name-only", "--diff-filter=AM", f"{base}...HEAD"],
        text=True,
    ).split()
    out = []
    for name in names:
        size = subprocess.check_output(
            ["git", "cat-file", "-s", f"HEAD:{name}"], text=True
        ).strip()
        out.append((name, int(size)))
    return out


def artifact_size(pointer_path: str) -> int:
    """Size the pointer refers to, which is the number that actually matters."""
    doc = yaml.safe_load(
        subprocess.check_output(["git", "show", f"HEAD:{pointer_path}"], text=True)
    )
    return int(doc["outs"][0].get("size", 0))


def main(base: str) -> int:
    budgets = load_budgets()
    failures, notes = [], []

    for path, blob_size in added_files(base):
        rule = budget_for(path, budgets)
        if rule is None:
            continue

        if path.endswith(".dvc"):
            size = artifact_size(path)
            limit = rule.get("artifact_mb")
            kind = "tracked artifact"
        else:
            size = blob_size
            limit = rule.get("blob_mb")
            kind = "git blob"

        if limit is None:
            continue

        if size > limit * MB:
            failures.append(
                f"{path}: {kind} is {size / MB:,.1f} MB, budget {limit} MB "
                f"(rule '{rule['path']}')"
            )
        elif size > 0.8 * limit * MB:
            notes.append(f"{path}: {size / MB:,.1f} MB, {size / (limit * MB):.0%} of budget")

    for note in notes:
        print(f"::notice::approaching budget — {note}")

    if failures:
        print("\nSize budget exceeded:\n", file=sys.stderr)
        for f in failures:
            print(f"  {f}", file=sys.stderr)
            print(f"::error::{f}")
        print(
            "\nIf this artifact belongs in the repository, track it with DVC rather "
            "than committing it. If the budget is genuinely wrong, raise it in "
            ".size-budgets.yml in its own commit so the change is reviewed.",
            file=sys.stderr,
        )
        return 1

    print(f"size budget: {len(added_files(base))} changed path(s) within budget")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1] if len(sys.argv) > 1 else "origin/main"))
# .size-budgets.yml — reviewed like code, because raising a limit is a decision
budgets:
  - path: "data/raw/"
    blob_mb: 1            # raw data is tracked, never committed directly
    artifact_mb: 8000
  - path: "data/interim/"
    blob_mb: 1
    artifact_mb: 2000
  - path: "scripts/"
    blob_mb: 2
  - path: "docs/"
    blob_mb: 10           # diagrams and screenshots
  - path: ""              # everything else
    blob_mb: 5

The warning at 80% is the part that changes behaviour. A budget that only speaks when it blocks a merge is a surprise; one that says “this artifact is at 84% of its budget” gives the team a chance to think about tiling before the day it fails.

Different directories, different limits A grid of four path prefixes against the blob budget and the artifact budget each carries, showing that raw data allows very large tracked artifacts while permitting almost nothing to be committed directly. Git blob Tracked artifact data/raw/ 1 MB 8 GB data/interim/ 1 MB 2 GB scripts/ 2 MB n/a docs/ 10 MB n/a The asymmetry in the first row is the whole rule: track it, do not commit it.

Validation & Output Verification

# The check must fail on a deliberately oversized blob
head -c 12000000 /dev/urandom > scripts/oversized.bin
git add scripts/oversized.bin && git commit -qm "test: oversized blob"
python scripts/check_size_budget.py origin/main \
  && echo "FAIL: budget did not catch a 12 MB blob in scripts/" \
  || echo "budget catches an oversized blob"
git reset -q --hard HEAD~1

# A large TRACKED artifact must pass where a raw blob would not
dvc add data/raw/orthomosaic.tif && git add data/raw/orthomosaic.tif.dvc
python scripts/check_size_budget.py origin/main && echo "tracked artifact allowed"
# Wire it into the pipeline as a cheap early gate
- name: Size budget
  run: python scripts/check_size_budget.py "origin/${{ github.base_ref }}"

Order it before the expensive geometry checks, per the fail-fast ordering in the parent guide: a 4 GB accidental commit should fail in under a second rather than after two minutes of topology validation.

Failure Modes

  • The check passes and history still grew — symptom: a large blob reaches the default branch. Root cause: the check compared against HEAD~1 rather than the merge base, so it missed additions from earlier commits on the branch. Fix: diff against the merge base with three-dot notation, as above.

  • Legitimate artifacts are blocked — symptom: the team disables the check. Root cause: one global limit applied to tracked artifacts and raw blobs alike. Fix: separate blob_mb from artifact_mb, and set the artifact budget from the real distribution.

  • The budget file is edited in the same commit as the violation — symptom: the limit rises whenever it is inconvenient. Root cause: no separation between the rule and the change. Fix: fail when .size-budgets.yml is modified in a commit that also adds data, and require the limit change on its own.

  • A pointer’s size field is absent — symptom: a tracked artifact measured as zero. Root cause: an older pointer format without a size field. Fix: treat a missing size as a failure to measure and report it, rather than as zero.

The cost of one accidental commit, over time A timeline of an accidentally committed four-gigabyte raster: it merges unnoticed, is discovered weeks later, and by then removing it would require a history rewrite that invalidates every clone — so it is usually left in place forever. Committed 4 GB, no check day 0 Merged review saw a filename day 0 Noticed clones are slow week 3 Left in place a rewrite breaks every clone week 3+ A check costing under a second at day zero replaces a decision nobody wants to make at week three.

Back to CI/CD Validation Pipelines for Spatial Repositories