Tuning Tolerance Thresholds for Conflict Detection

Every conflict detector rests on a handful of numbers, and those numbers are usually inherited from whatever the first implementation used — which means nobody can say what they cost. This page is a focused companion to automated conflict detection in merge requests.

Concept & Context

A detector reports a conflict when a measurement crosses a threshold: overlap area above so many square metres, displacement above so many centimetres, attribute divergence on a sensitive column. Set them too tight and reviewers drown in alerts about capture noise, learn that alerts are noise, and start approving without looking. Set them too loose and real conflicts merge silently.

Both failures are gradual and neither announces itself. What makes them tractable is that the answer already exists in the audit trail: every conflict a steward resolved carries a decision, and that decision is a label. A year of resolutions is a corpus, and a corpus turns threshold selection from an argument into a measurement.

The output of tuning is not one number but a curve — how many real conflicts are missed and how many false alarms are raised at each candidate value. Choosing a point on that curve is a policy decision, because the two errors have very different costs, and on a cadastral layer they are not within an order of magnitude of each other.

Core Algorithmic Pipeline

  1. Extract labelled cases from resolved merges: the measurement at detection time and the steward’s verdict.
  2. Split by layer group, since capture accuracy and error cost both vary.
  3. Sweep each threshold across a plausible range, recording missed conflicts and false alarms.
  4. Choose a point using the cost ratio the team can defend, not a symmetric score.
  5. Record the chosen values with the evidence, and re-measure on a schedule.
Where the labels come from A chain of four stages turning past work into a tuning corpus: resolved merges carry steward decisions, each decision plus the measurement at detection time becomes a labelled case, cases are grouped by layer, and the sweep produces the curve. Audit trail resolved conflicts decisions Labelled case measurement + verdict grouped Layer group same capture accuracy swept Sweep misses vs alarms The corpus is a by-product of work the team already did, which is why tuning costs almost nothing.

Working Implementation

"""Tune conflict-detection thresholds against decisions already made."""
from __future__ import annotations

import json
from dataclasses import dataclass

import numpy as np


@dataclass(frozen=True)
class Case:
    """One historical detection with the verdict a steward gave it."""
    layer_group: str
    overlap_m2: float
    displacement_m: float
    was_real_conflict: bool


def load_cases(path: str) -> list[Case]:
    """Read the audit trail. Every resolved conflict is one labelled case."""
    return [Case(**row) for row in json.load(open(path, encoding="utf-8"))]


def sweep(cases: list[Case], attribute: str, candidates: np.ndarray) -> list[dict]:
    """Missed conflicts and false alarms at each candidate threshold."""
    values = np.array([getattr(c, attribute) for c in cases])
    labels = np.array([c.was_real_conflict for c in cases])

    rows = []
    for t in candidates:
        flagged = values >= t
        missed = int(np.sum(labels & ~flagged))          # real, not flagged
        false_alarms = int(np.sum(~labels & flagged))    # flagged, not real
        caught = int(np.sum(labels & flagged))
        rows.append({
            "threshold": float(t),
            "missed": missed,
            "false_alarms": false_alarms,
            "caught": caught,
            "recall": caught / max(int(labels.sum()), 1),
            "precision": caught / max(caught + false_alarms, 1),
        })
    return rows


def choose(rows: list[dict], miss_cost: float, alarm_cost: float) -> dict:
    """Pick the threshold minimising total cost under the team's own weighting.

    miss_cost is what it costs when a real conflict merges unnoticed; alarm_cost
    is what a spurious review costs. On a cadastral layer the ratio is commonly
    50:1 or worse, and stating it explicitly is the point of this function.
    """
    for row in rows:
        row["cost"] = row["missed"] * miss_cost + row["false_alarms"] * alarm_cost
    return min(rows, key=lambda r: (r["cost"], r["threshold"]))


def report(cases: list[Case], group: str) -> dict:
    subset = [c for c in cases if c.layer_group == group]
    if len(subset) < 30:
        raise ValueError(
            f"{group}: only {len(subset)} labelled case(s) — too few to tune on. "
            "Keep the current thresholds and revisit when the corpus grows."
        )

    overlap = sweep(subset, "overlap_m2", np.geomspace(0.01, 100, 40))
    displacement = sweep(subset, "displacement_m", np.geomspace(0.005, 5, 40))

    chosen = {
        "overlap_m2": choose(overlap, miss_cost=50, alarm_cost=1),
        "displacement_m": choose(displacement, miss_cost=50, alarm_cost=1),
        "corpus_size": len(subset),
    }

    print(f"\n{group}  ({len(subset)} labelled cases)")
    print(f"{'threshold':>12} {'missed':>8} {'alarms':>8} {'recall':>8} {'precision':>10}")
    for row in overlap[::4]:
        print(f"{row['threshold']:>12.3f} {row['missed']:>8d} "
              f"{row['false_alarms']:>8d} {row['recall']:>8.2f} {row['precision']:>10.2f}")
    print(f"chosen overlap threshold: {chosen['overlap_m2']['threshold']:.3f} m²")
    return chosen

A representative sweep on a cadastral corpus:

cadastral  (412 labelled cases)
   threshold   missed   alarms   recall  precision
       0.010        0      291     1.00       0.24
       0.046        0      168     1.00       0.35
       0.215        1       47     0.99       0.66
       1.000        6        9     0.94       0.91
       4.642       23        1     0.75       0.99
      21.544       61        0     0.34       1.00
chosen overlap threshold: 0.215 m²

The chosen point keeps recall at 99% and cuts alerts by six sevenths relative to the tightest setting. Moving one step further loses six real conflicts to save 38 reviews — a trade nobody would take on a cadastral layer, and one that might be right for land cover.

The trade-off, read off a real corpus Horizontal bars showing false alarms raised at four candidate overlap thresholds on a labelled cadastral corpus, annotated with the real conflicts each threshold misses — from 291 alarms and zero misses at the tightest setting to one alarm and 23 misses at the loosest. FALSE ALARMS PER THRESHOLD (412 LABELLED CASES) 0.01 m² 291 0 missed — and reviewers stop reading 0.22 m² 47 1 missed — the chosen point 1.0 m² 9 6 missed 4.6 m² 1 23 missed — unacceptable on cadastral data Choosing a point on this curve is a policy decision, because the two errors do not cost the same.

Validation & Output Verification

# The corpus must be balanced enough to tune on
from collections import Counter
cases = load_cases("audit/resolved_conflicts.json")
counts = Counter((c.layer_group, c.was_real_conflict) for c in cases)
for group in {c.layer_group for c in cases}:
    real = counts[(group, True)]
    noise = counts[(group, False)]
    print(f"{group:<14} {real:>4} real, {noise:>4} noise")
    assert min(real, noise) >= 10, f"{group}: too one-sided to tune"

# The chosen thresholds must beat the incumbent on the same corpus
import yaml
current = yaml.safe_load(open("config/conflict-thresholds.yml"))
for group, chosen in tuned.items():
    old = current[group]["overlap_m2"]
    old_row = next(r for r in sweep(cases, "overlap_m2", [old]))
    assert chosen["overlap_m2"]["cost"] <= old_row["cost"], (
        f"{group}: tuned threshold is not better than the current {old}"
    )
# Thresholds are configuration, so they belong under review like code
git log --oneline -- config/conflict-thresholds.yml | head
# A change should be accompanied by the sweep output that justified it.

Schedule the re-measurement. Capture equipment improves, editing practice changes, and a threshold tuned two years ago is describing a workflow that no longer exists.

Failure Modes

  • Reviewers approve without looking — symptom: median review time under a minute. Root cause: thresholds too tight, so most alerts are noise. Fix: sweep, and move to a point with defensible precision; the parent guide’s gate ordering keeps the cheap checks first regardless.

  • A real conflict merged silently — symptom: a dispute about geometry nobody reviewed. Root cause: thresholds loosened to reduce complaints rather than by measurement. Fix: re-tune with the real cost ratio, and record it in the configuration.

  • Tuning overfits — symptom: thresholds that look perfect on the corpus and behave badly in production. Root cause: a corpus dominated by one incident. Fix: check the corpus balance, and hold out a slice by date rather than at random so the evaluation reflects a later period.

  • One threshold for every layer — symptom: cadastral alerts too coarse and land-cover alerts too noisy. Root cause: layer groups not separated. Fix: tune per group and record the group in each layer’s metadata.

Reading the symptoms of a mistuned gate Two panels. Thresholds set too tight produce fast approvals, a permanently full queue and batching behaviour; thresholds set too loose produce unreviewed corrections, gaps in the audit trail and disputes raised from outside the team. TOO TIGHT Median review time under a minute The queue is never empty and nobody tracks its age Editors batch changes to amortise the wait Approval stops carrying information TOO LOOSE Boundary corrections in history with no review Gaps in the audit trail on the interesting dates Thresholds last changed to stop complaints First evidence is a dispute from outside the team Both are measurable: track approval time and the distribution of tiers, monthly. Neither failure announces itself, and both look like a quiet queue from the outside.

Back to Automated Conflict Detection in Merge Requests