Automating DVC Push on Merge with GitLab CI

This pipeline waits for a merge to the default branch, materializes the DVC-managed spatial artifacts, revalidates them on a clean runner, pushes the validated bytes to the shared remote, and tags the resulting data version. It is a focused companion to CI/CD validation pipelines for spatial repositories.

Concept & Context

Merge-request pipelines validate a change; they do not promote it. When a spatial pull request is approved and merged, two things still need to happen that GitLab does not do on its own: the reviewed artifact bytes must land on the canonical DVC remote, and the exact pairing of code and data must be pinned so it can be reproduced. Doing this from feature branches is a mistake — it fills the remote with unreviewed, possibly invalid versions. The promotion belongs to a pipeline that runs only after the merge lands on main.

That post-merge job re-runs the same validation the merge request already passed, because a clean runner is the only environment nobody can bypass, and because a merge can combine two individually valid branches into a jointly invalid state. The artifact model this depends on — pointer files in Git, bytes on a remote — is the subject of large file handling in DVC for GIS; this page assumes that model and concentrates on the promote-and-tag mechanics, including the credential and cache handling that make it safe and fast.

Core Promotion Steps

The pipeline runs five steps, each expressed as a stage or a scripted command in the .gitlab-ci.yml below.

  1. Scope to the default branch. A rules clause restricts the promote job to pushes on main, so it never fires on merge-request pipelines or feature branches.
  2. Restore cache and credentials. The .dvc/cache directory is cached across runs, and the remote credentials are injected from masked, protected CI/CD variables that only protected branches can read.
  3. Pull and validate. dvc pull materializes the artifacts referenced by the merge commit, then the shared validation script runs and aborts the job on any critical finding.
  4. Push to the remote. dvc push persists the validated artifact bytes to the canonical remote so every collaborator can reproduce them.
  5. Tag the data version. An annotated git tag is created against the merge commit and pushed, freezing the code-plus-data pairing under a named release.

Working Implementation

The entire pipeline lives in one .gitlab-ci.yml. The validate stage guards the promotion; the promote stage only runs if validation succeeds, because GitLab stages execute in order and a failed job halts the pipeline.

# .gitlab-ci.yml — validate then promote DVC-managed spatial data on merge to main
stages:
  - validate
  - promote

default:
  image: python:3.11-slim
  # Credentials come from masked, protected CI/CD variables (Settings > CI/CD).
  # These are exposed only to pipelines running on the protected `main` branch.
  variables:
    AWS_ACCESS_KEY_ID: $DVC_REMOTE_KEY_ID
    AWS_SECRET_ACCESS_KEY: $DVC_REMOTE_SECRET
  cache:
    key: "dvc-cache-$CI_COMMIT_REF_SLUG"
    paths:
      - .dvc/cache
  before_script:
    - pip install --quiet -r requirements.txt   # pinned geopandas, dvc, etc.
    - dvc --version

# ---- Stage 1: revalidate the merged state on a clean runner ----
revalidate:
  stage: validate
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH   # only on merge to main
  script:
    - dvc pull --quiet                              # materialize artifact bytes
    - |
      # Guard: fail loudly if pull restored pointers instead of real data
      python - <<'PY'
      import glob, os, sys
      layers = glob.glob("data/**/*.gpkg", recursive=True)
      empty = [p for p in layers if os.path.getsize(p) < 1024]
      if empty:
          print(f"::DVC pull incomplete, placeholder files: {empty}")
          sys.exit(1)
      PY
    - python validate_spatial.py $(git ls-files '*.gpkg' '*.geojson')

# ---- Stage 2: push validated bytes to the remote and tag the version ----
promote:
  stage: promote
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    # Push the validated artifacts to the canonical DVC remote
    - dvc push --quiet
    # Configure git identity for the tagging commit
    - git config user.email "ci@geospatial-data-versioning.com"
    - git config user.name "GitLab CI"
    # Derive a reproducible, sortable data-version tag
    - export TAG="data-v$(date +%Y.%m.%d)-${CI_COMMIT_SHORT_SHA}"
    - |
      git tag -a "$TAG" -m "Validated spatial data promoted on merge
      commit: $CI_COMMIT_SHA
      pipeline: $CI_PIPELINE_ID"
    # Push the tag over an authenticated HTTPS remote using a project token.
    # CI_DATA_TAG_TOKEN is a masked, protected project access token with
    # write_repository scope; the job token cannot push tags.
    - git remote add tagging "https://oauth2:${CI_DATA_TAG_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git"
    - git push tagging "$TAG"
    - echo "Promoted and tagged spatial data version $TAG"
  environment:
    name: production-data

Three parts of this file carry the real weight. The rules clause on both jobs — $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH — is what confines promotion to the post-merge pipeline. The cache block keyed on the ref slug keeps .dvc/cache warm so dvc pull restores unchanged artifacts locally instead of over the network. And the inline python guard after dvc pull refuses to proceed if the pull left placeholder files behind, which is the single most common way a promotion silently pushes nothing useful.

Validation & Output Verification

Prove the pipeline promoted real, reproducible data rather than appearing to succeed.

  • Confirm the remote received the bytes. After a merge, run dvc status --cloud on any checkout of the tag; a clean report (“Cache and remote are in sync”) confirms dvc push uploaded every hash the pointer files reference.
  • Reproduce from the tag. In a fresh clone, git checkout data-v2026.07.11-<sha> && dvc pull must restore byte-identical layers. Verify with dvc status reporting no changes and, for a hard check, compare a layer’s feature count and geometry hash against the pre-merge value.
  • Verify the tag is annotated and pushed. git show <tag> should display the annotation with the commit SHA and pipeline ID, and git ls-remote --tags origin should list it.
  • Check the guard fires. Temporarily point the remote at an unreachable endpoint and confirm the revalidate stage exits non-zero on the placeholder check rather than pushing an empty version.

Failure Modes

Symptom: the pipeline succeeds but dvc status --cloud later reports missing hashes. Root cause: dvc push ran before all artifacts were materialized, or a partial cache masked missing files. Fix: keep the placeholder guard after dvc pull and let it abort the job before promote runs.

Symptom: dvc push fails with an access-denied error. Root cause: the remote credentials were stored as unprotected CI/CD variables and are not exposed to the protected main branch pipeline. Fix: mark the variables both masked and protected so the promote pipeline can read them.

Symptom: git push tagging fails with 403 Forbidden. Root cause: the default CI_JOB_TOKEN lacks permission to write tags. Fix: use a project or group access token with write_repository scope, stored as a masked protected variable, as shown in the pipeline.

Symptom: the promote job runs on a feature branch. Root cause: the rules clause is missing or compares against the wrong variable. Fix: gate both jobs on $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH so only merges to the default branch trigger promotion.

Back to CI/CD Validation Pipelines for Spatial Repositories