Reconciling Offline Edits from Field Collection Apps

A returned collection package is not a file to copy over the trunk: it is a log of things a crew observed, which has to be replayed against a trunk that moved while they were out. This page is a focused companion to offline field collection and sync reconciliation.

Concept & Context

Field collection applications keep a change log. QField’s synchronisation works from a copy of the project with change tracking on the GeoPackage; Esri’s Field Maps offline areas maintain a replica with a per-feature edit record. Both were designed for exactly this situation, and both are routinely thrown away by a sync process that copies the returned container over the source layer.

Reading the log instead of the final state preserves two things a diff destroys. The first is deletion intent: a feature absent from the returned container might have been deliberately retired, or might have been outside the crew’s extent and never present. The second is grouping: a parcel split is one field decision that touches three rows, and a reviewer looking at three unrelated row changes cannot see it.

Everything that follows assumes the package carries its base revision and that identifiers were minted on the device — the two prerequisites from the parent guide. Without them, reconciliation degrades to a two-way overwrite regardless of how carefully the rest is written.

Core Algorithmic Pipeline

1. Validate the package. Base revision present and reachable, CRS matching the project, schema matching what went out, identifiers conforming to the crew prefix. Anything failing goes to quarantine untouched.
2. Extract operations from the container’s change log, ordered by observation time rather than by row id.
3. Classify each operation against the base revision and the current trunk: applicable, superseded, or conflicting.
4. Apply the applicable set on a branch, in one transaction, recording the package and operation identifiers as applied.
5. Report per-feature outcomes to the crew, naming what needs their notes.

Working Implementation

"""Reconcile a returned GeoPackage collection package against the current trunk."""
from __future__ import annotations

import json
import sqlite3
import subprocess
from pathlib import Path

import geopandas as gpd


def validate_package(pkg_dir: Path, expected_crs: str, crew_prefix: str) -> dict:
    """Refuse anything that cannot be reconciled safely, before touching the trunk."""
    meta = json.loads((pkg_dir / "package.json").read_text())

    base = meta.get("base_revision")
    if not base:
        raise ValueError("package records no base revision")
    if subprocess.run(["git", "cat-file", "-e", f"{base}^{{commit}}"],
                      capture_output=True).returncode != 0:
        raise ValueError(f"base revision {base[:12]} is not reachable in this repository")

    layer = gpd.read_file(pkg_dir / "collect.gpkg")
    if layer.crs is None or layer.crs.to_string() != expected_crs:
        raise ValueError(f"package CRS {layer.crs} does not match project {expected_crs}")

    created = layer[layer["parcel_id"].astype(str).str.startswith(crew_prefix)]
    stray = set(layer["parcel_id"]) - set(created["parcel_id"])
    unknown_new = {p for p in stray if str(p).count("-") == 1
                   and not str(p).startswith(crew_prefix)}
    if unknown_new:
        raise ValueError(f"identifiers minted with a foreign prefix: {sorted(unknown_new)[:5]}")

    return meta


def read_operations(gpkg: Path, table: str = "parcels") -> list[dict]:
    """Read the app's change log, newest last, ordered by observation time.

    QField writes an audit table alongside the layer when change tracking is on;
    the column names below match the default configuration.
    """
    con = sqlite3.connect(gpkg)
    con.row_factory = sqlite3.Row
    rows = con.execute(
        f"""
        SELECT op_id, operation, feature_id, observed_at, fix_mode, hrms_m,
               editor, reason
          FROM {table}_changes
         ORDER BY observed_at ASC, op_id ASC
        """
    ).fetchall()
    con.close()
    return [dict(r) for r in rows]


def classify(operations, base_state, trunk_state, applied_ids):
    """Split operations into applicable, already-applied, and conflicting."""
    applicable, skipped, conflicts = [], [], []

    for op in operations:
        if op["op_id"] in applied_ids:
            skipped.append(op)                       # idempotent re-run
            continue

        fid = op["feature_id"]
        base = base_state.get(fid)
        trunk = trunk_state.get(fid)

        if op["operation"] == "create":
            (conflicts if fid in trunk_state else applicable).append(
                (op, "identifier already on trunk") if fid in trunk_state else op
            )
        elif trunk is None:
            conflicts.append((op, "feature no longer on trunk"))
        elif base is not None and base["fingerprint"] == trunk["fingerprint"]:
            applicable.append(op)
        else:
            conflicts.append((op, "changed on both sides while offline"))

    return applicable, skipped, conflicts


def reconcile(pkg_dir: Path, layer_path: Path, expected_crs="EPSG:3035",
              crew_prefix="EAST-") -> dict:
    meta = validate_package(pkg_dir, expected_crs, crew_prefix)
    operations = read_operations(pkg_dir / "collect.gpkg")

    base_state = fingerprint_at(layer_path, meta["base_revision"])
    trunk_state = fingerprint_at(layer_path, "HEAD")
    applied_ids = load_applied(meta["package_id"])

    applicable, skipped, conflicts = classify(
        operations, base_state, trunk_state, applied_ids
    )

    branch = f"sync/{meta['crew_prefix'].rstrip('-').lower()}-{meta['cut_at'][:10]}"
    subprocess.run(["git", "switch", "-c", branch], check=True)
    apply_operations(layer_path, pkg_dir / "collect.gpkg", applicable)
    record_applied(meta["package_id"], [op["op_id"] for op in applicable])

    return {
        "branch": branch,
        "applied": len(applicable),
        "already_applied": len(skipped),
        "conflicts": [
            {"feature": op["feature_id"], "observed_at": op["observed_at"],
             "reason": reason, "editor": op["editor"]}
            for op, reason in conflicts
        ],
    }

The applied_ids set is what makes a second attempt safe. Crews hand in the same package twice more often than anyone expects — after a failed upload, after a support call, after a laptop swap — and a reconciliation that duplicates every edit on the second attempt is a much worse afternoon than one that reports “0 applied, 214 already applied”.

Validation & Output Verification

Check the reconciliation before the branch goes anywhere:

# Nothing outside the crew's assigned extent may have been touched
python - <<'PY'
import json, geopandas as gpd
from shapely.geometry import box
meta = json.load(open("packages/crew-east/package.json"))
extent = box(*meta["extent"])
changed = gpd.read_file("data/parcels.gpkg").query("parcel_id in @touched_ids")
outside = changed[~changed.geometry.within(extent.buffer(50))]
assert outside.empty, f"{len(outside)} feature(s) edited outside the assigned extent"
print("all edits inside the assigned extent")
PY

# The normal gates apply — a field package earns no exemption
python scripts/validate_spatial.py data/parcels.gpkg --crs EPSG:3035 --topology

# Every applied operation must appear exactly once in the applied log
sqlite3 sync.db "SELECT op_id, count(*) c FROM applied_ops GROUP BY op_id HAVING c > 1"
# expected: no rows

Then produce the report the crew actually reads:

crew-east package 2026-07-15  (base 4f2a91c, 3 weeks out)
  applied         214
  already applied   0
  conflicts         6

  EAST-7K3M2P9Q4R  changed on both sides while offline   (obs 2026-07-02, RTK 0.02 m)
  EAST-4A9B1C2D3E  feature no longer on trunk            (obs 2026-07-04, DGPS 0.6 m)
  …

Naming the features is the point. A crew told “six conflicts” cannot help; a crew told which six will bring the field book to the review.

Failure Modes

• The sync reverted trunk edits inside the extent — symptom: work done by others while the crew was out disappears. Root cause: the returned container copied over the layer. Fix: replay the change log; refuse packages whose change table is missing or empty.

• Deletions arrive as absences — symptom: features the crew never saw are retired. Root cause: deletion inferred from a diff of final state. Fix: apply only explicit retire operations from the log.

• A rerun duplicated every edit — symptom: doubled features after a second attempt. Root cause: no applied-operation ledger. Fix: record package and operation identifiers, and skip what is already applied.

• Everything conflicts — symptom: the conflict list is as long as the operation list. Root cause: the base revision is unreachable, so classification fell back to comparing against the trunk. Fix: extend history retention beyond the longest deployment, and check reachability during validation as above.

Related

• Offline Field Collection and Sync Reconciliation — the parent guide, including the package format this reads
• Designing Conflict-Free Identifiers for Offline Capture — the identifier scheme validation depends on
• Automating Attribute Reconciliation with Pandas and GeoPandas — resolving the attribute half of a conflicting operation
• Manual Review Triggers for Critical Edits — where the conflicts in the report are routed

Back to Offline Field Collection and Sync Reconciliation