Estimating Storage Growth for Repeat LiDAR Surveys

The question that decides whether a point-cloud repository gets built is “how big will this get in five years”, and it is usually answered by multiplying — which overestimates by roughly an order of magnitude and kills the project. This page is a focused companion to point-cloud versioning and branching strategies.

Concept & Context

A repeat LiDAR programme flies the same area on a cadence: annually for a monitoring network, more often over active slopes or subsidence areas. Each survey produces a full point cloud, and the naive projection multiplies its size by the number of epochs planned.

That model is wrong because most of the ground does not move. Under tile-level content addressing, a tile whose points are unchanged hashes identically and stores nothing at all. The real growth is driven by the fraction of tiles that genuinely differ, and by how well the differences compress once separated into streams.

Two decisions dominate the result, and both are made early. Tile size sets how much unchanged ground gets dragged into a changed tile — halve the tile edge and a localised change touches roughly a quarter of the area it did. Whether reclassification is committed as a data change decides whether an epoch that surveyed nothing can still cost a full resurvey, which is the single largest avoidable line in most projections.

The projection that kills the project, beside the real one Horizontal bars comparing a naive ten-epoch projection that multiplies survey size by epoch count against measured projections at three tile sizes, showing the naive figure is roughly ten times the real one. STORAGE AFTER TEN ANNUAL EPOCHS Naive (snapshot per epoch) 8316 GB assumes nothing is ever unchanged Measured, 1000 m tiles 1265 GB Measured, 500 m tiles 991 GB Measured, 250 m tiles 846 GB The gap between the first bar and the rest is the whole reason the programme looks unaffordable on paper.

Core Algorithmic Pipeline

  1. Measure the tile change rate between two real epochs at the candidate tile size.
  2. Measure the delta ratio on the changed tiles: stored bytes divided by full-tile bytes.
  3. Project forward across the planned cadence and retention window.
  4. Vary tile size and re-measure, since the change rate is a function of it.
  5. Model reclassification separately, because it changes everything at once.
Where each model input comes from A chain of four stages: two real epochs give a tile change rate, the deltas between them give a delta ratio, the survey cadence gives the epoch count, and the three together produce the growth curve. Two epochs manifests, hashed per tile compared Change rate fraction of tiles differing measured Delta ratio stored ÷ full, on changed tiles projected Curve projected over the cadence Every input is measured on real data; nothing in the model is a rule of thumb.

Working Implementation

"""Project storage growth for a versioned repeat-survey point-cloud repository."""
from __future__ import annotations

from dataclasses import dataclass


@dataclass
class Measured:
    """Numbers taken from two real epochs, not assumed."""
    tiles: int
    tile_bytes_mean: float          # bytes per tile, full encoding
    change_rate: float              # fraction of tiles whose contents differ
    delta_ratio: float              # stored bytes / full bytes, on a changed tile
    reclass_rate: float = 0.0       # fraction of epochs including a reclass pass


def measure(epoch_a_manifest: dict, epoch_b_manifest: dict,
            delta_sizes: dict[str, int]) -> Measured:
    """Derive the model's inputs from two manifests and the deltas between them."""
    common = set(epoch_a_manifest) & set(epoch_b_manifest)
    changed = [t for t in common if epoch_a_manifest[t]["hash"] != epoch_b_manifest[t]["hash"]]

    full_mean = sum(epoch_b_manifest[t]["bytes"] for t in common) / len(common)
    if changed:
        stored = sum(delta_sizes[t] for t in changed)
        full = sum(epoch_b_manifest[t]["bytes"] for t in changed)
        ratio = stored / full
    else:
        ratio = 0.0

    return Measured(
        tiles=len(common),
        tile_bytes_mean=full_mean,
        change_rate=len(changed) / len(common),
        delta_ratio=ratio,
    )


def project(m: Measured, epochs: int) -> list[dict]:
    """Cumulative stored bytes after each epoch, including the baseline."""
    baseline = m.tiles * m.tile_bytes_mean
    rows = [{"epoch": 0, "added_gb": baseline / 1e9, "total_gb": baseline / 1e9}]

    total = baseline
    for e in range(1, epochs + 1):
        changed_tiles = m.tiles * m.change_rate
        added = changed_tiles * m.tile_bytes_mean * m.delta_ratio

        # A reclassification pass rewrites every tile's contents, so the epoch
        # costs a full snapshot's worth of deltas regardless of ground movement.
        if m.reclass_rate and (e % max(round(1 / m.reclass_rate), 1) == 0):
            added = m.tiles * m.tile_bytes_mean * m.delta_ratio
            note = "reclassification pass"
        else:
            note = ""

        total += added
        rows.append({"epoch": e, "added_gb": added / 1e9,
                     "total_gb": total / 1e9, "note": note})
    return rows


def compare_tile_sizes(measurements: dict[int, Measured], epochs: int) -> None:
    """The tile-size decision, shown as the curve it actually produces."""
    print(f"{'tile (m)':>9} {'change rate':>12} {'delta ratio':>12} "
          f"{'total after ' + str(epochs) + ' epochs (GB)':>32}")
    for edge, m in sorted(measurements.items()):
        total = project(m, epochs)[-1]["total_gb"]
        print(f"{edge:>9} {m.change_rate:>12.1%} {m.delta_ratio:>12.2f} {total:>32.1f}")

A worked projection for an annual programme over 4,200 tiles averaging 180 MB:

 tile (m)  change rate  delta ratio    total after 10 epochs (GB)
      250         6.1%         0.19                         846.3
      500        14.8%         0.21                         991.4
     1000        31.2%         0.22                        1264.7

naive projection (full snapshot per epoch):                 8316.0

Two things stand out. Versioning costs about a tenth of the naive estimate, which is the number that decides whether the programme is affordable. And halving the tile edge from 1000 m to 500 m removes more than 270 GB over the decade — a decision made once, at the start, worth more than any later tuning.

The reclassification line is worth modelling explicitly:

without = Measured(tiles=4200, tile_bytes_mean=180e6, change_rate=0.061,
                   delta_ratio=0.19)
with_reclass = Measured(tiles=4200, tile_bytes_mean=180e6, change_rate=0.061,
                        delta_ratio=0.19, reclass_rate=0.33)   # every third epoch

print(f"no reclassification: {project(without, 10)[-1]['total_gb']:.0f} GB")
print(f"reclass every 3rd:   {project(with_reclass, 10)[-1]['total_gb']:.0f} GB")
# no reclassification: 846 GB
# reclass every 3rd:   2143 GB

Three reclassification passes cost more than the entire decade of actual ground change. That is the case for committing classification as a separate, clearly labelled change — a point the classification-aware diff makes from the review side.

Validation & Output Verification

# Ground the model on two real epochs before trusting any projection
python scripts/measure_growth.py \
  --epoch-a manifests/2025-03.json \
  --epoch-b manifests/2026-03.json \
  --deltas deltas/2025-03_to_2026-03/ | tee growth-inputs.json

# The change rate must be plausible: total change should track surveyed change
jq -r '.change_rate' growth-inputs.json
# A stable monitoring area reporting 80% changed tiles means tiling or point
# ordering changed, not the ground — check before modelling anything.
# Re-measure against reality after each epoch; a model nobody checks drifts
import json
predicted = json.load(open("growth-projection.json"))
actual = json.load(open("storage-actuals.json"))

for epoch in sorted(set(predicted) & set(actual), key=int):
    p, a = predicted[epoch]["total_gb"], actual[epoch]["total_gb"]
    drift = abs(a - p) / p
    flag = "  <-- re-measure inputs" if drift > 0.25 else ""
    print(f"epoch {epoch}: predicted {p:7.1f} GB, actual {a:7.1f} GB "
          f"({drift:+.0%}){flag}")

Failure Modes

  • The projection kills the project — symptom: a five-year estimate an order of magnitude too large. Root cause: multiplying survey size by epochs, which assumes full snapshots. Fix: measure the tile change rate; it is usually under 15% on a stable area.

  • Actual growth far exceeds the model — symptom: an epoch costing a full snapshot. Root cause: an uncommitted reclassification pass, or a retiling. Fix: model reclassification explicitly, and never retile without treating it as a migration.

  • Change rate reported near 100% — symptom: every tile differs after a routine resurvey. Root cause: non-canonical point ordering or a changed compression setting, so hashes differ without content differing. Fix: hash content rather than bytes, per the canonical ordering in the parent guide.

  • Tile size chosen for convenience — symptom: growth dominated by unchanged ground dragged along by changed tiles. Root cause: tile size set by the delivery format rather than measured. Fix: measure the change rate at two or three candidate sizes before the first epoch is committed.

What a reclassification pass costs a decade A timeline of ten annual epochs in which three reclassification passes each rewrite every tile's contents, so those three epochs together cost more than the entire decade of actual ground movement. Ground change only 6% of tiles per epoch yr 1-3 Reclassification every tile rewritten yr 3 Reclassification every tile again yr 6 Reclassification and again yr 9 846 GB without them, 2143 GB with — three parameter changes cost more than ten years of surveying.

Back to Point-Cloud Versioning and Branching Strategies