Geospatial Portal

Promoting GeoNode Releases with Argo CD

This guide promotes a portal release between environments as a reviewed change to a repository, with the ordering that a stateful geospatial stack requires and a rollback path that works when the pipeline does not. It belongs to Helm vs Kustomize for GeoNode Deployments, within the Infrastructure Orchestration & Configuration Management framework.

Prerequisites

  • Argo CD installed with access to the manifest repository and to both clusters or namespaces.
  • Images published by digest rather than by a moving tag, per the parity model in environment parity in geospatial CI pipelines.
  • Secrets handled outside the manifests, per sealing GeoNode secrets for GitOps.
  • A staging environment whose data volume is close enough to production that a migration’s duration is informative.

Promotion is a commit, not a deployment action

The property worth preserving is that every environment’s state is a readable point in the repository’s history. A promotion that is performed by a pipeline action leaves the repository describing the previous state and the cluster running the new one, which removes the main benefit of the model.

The same digest, promoted by a second commit A change merged to staging, verified, then the identical digest committed to the production overlay, with the reconciler applying each. merge to staging image digest pinned reconciler applies staging, automatically verify gates, and a real look commit to production the same digest, reviewed What this buys Every environment's running state is a commit. “What is in production?” is a file, not a cluster query, and “what changed on the 14th?” is a diff. Rolling back is reverting a specific commit rather than reconstructing one.

Step-by-step implementation

1. Define the applications 

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: geoportal-production
  namespace: argocd
spec:
  project: geoportal
  source:
    repoURL: https://github.com/agency/geoportal-manifests.git
    targetRevision: main
    path: clusters/production/geoportal
  destination:
    server: https://kubernetes.default.svc
    namespace: geoportal
  syncPolicy:
    # Production syncs on a human decision; staging can be automatic.
    automated: null
    syncOptions:
      - CreateNamespace=false
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 3
      backoff: {duration: 30s, factor: 2, maxDuration: 5m}

2. Order the sync so the migration runs at the right moment

A geospatial portal has a strict ordering requirement that a naive apply ignores: the database migration must complete before the application starts, and the search index rebuild must run after both.

# Sync waves express the order. Lower numbers run first, and a wave does not
# start until the previous one is healthy.
apiVersion: batch/v1
kind: Job
metadata:
  name: geonode-migrate
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 0            # a failed migration must stop the sync, not retry blindly
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: ghcr.io/agency/geonode@sha256:9f2c...
          command: ["python3", "manage.py", "migrate", "--noinput"]
---
# The application itself, after the migration.
# metadata.annotations: argocd.argoproj.io/sync-wave: "2"
---
# The index rebuild, after the application is healthy.
# metadata.annotations: argocd.argoproj.io/sync-wave: "3"
Sync waves for a stateful portal Migration, then application rollout, then index rebuild, each gated on the previous wave being healthy. wave 1 · migration pre-sync hook, no retry a failure stops the sync wave 2 · application portal, workers, renderer rolling, health-gated wave 3 · index rebuild after the app is healthy never from partial data Without waves, everything applies at once: the application can start against an unmigrated schema, and the index can be rebuilt while the migration is still running — both fail in ways that look like application bugs. Set backoffLimit to zero on the migration: a half-applied schema retried blindly is worse than a stopped sync.

3. Make migrations backward compatible so rollback is possible 

# A migration that adds a nullable column and backfills separately can be rolled
# back by reverting the application alone. One that renames a column cannot.
class Migration(migrations.Migration):
    atomic = False                      # long backfills should not hold one transaction
    operations = [
        migrations.AddField(
            model_name="dataset",
            name="publication_state",
            field=models.CharField(max_length=32, null=True),   # nullable first
        ),
        # Backfill in a separate, resumable step — not in this migration.
    ]

4. Promote by writing the digest 

# The promotion is a commit that copies the verified digest across.
DIGEST=$(yq '.images[0].newTag' clusters/staging/geoportal/kustomization.yaml)
yq -i ".images[0].newTag = \"$DIGEST\"" clusters/production/geoportal/kustomization.yaml
git add clusters/production/geoportal/kustomization.yaml
git commit -m "promote geonode $DIGEST to production"

Rolling back when the pipeline is part of the problem

A revert-and-reconcile rollback is clean and it depends on the repository, the runner and the reconciler all working. During the class of incident where a bad release and a broken pipeline coincide — which is not rare, because both often follow the same change — a second path is needed.

Two rollback paths, and when each is available Revert-and-reconcile as the primary path, and a direct in-cluster rollback as the fallback when the pipeline is unavailable. primary · revert the commit clean, auditable, and the repository stays the source of truth requires: repository, runner, reconciler use this whenever it is available fallback · roll back in cluster available from a laptop with cluster access alone creates drift — reconcile it back rehearse it, so it is not improvised 
# Fallback: suspend reconciliation, roll back, then repair the repository.
argocd app set geoportal-production --sync-policy none
kubectl -n geoportal rollout undo deploy/geonode
# ... and afterwards, revert the commit so the repository matches reality again.

Suspending automated sync first matters: a reconciler that is still enforcing the repository will undo the manual rollback within a minute, which during an incident reads as the platform fighting back.

One further practice makes releases easier to reason about over time: keep a short release note in the same commit that promotes the digest, saying what changed and what to watch. It costs a paragraph and it is what turns the repository history into an answer to “what changed around the time this started” — a question that arrives weeks later, from somebody who was not involved, about a symptom nobody connected to a release at the time.

Verification 

# 1. The two environments differ only by the intended fields
diff <(kustomize build clusters/staging/geoportal) <(kustomize build clusters/production/geoportal) \
  | grep -E '^[<>]' | grep -vE 'namespace|host|replicas|storage'
#   expect: no output beyond the declared differences

# 2. Waves ran in order
argocd app get geoportal-production --show-operation | grep -E 'Wave|Phase'
#   expect: wave 1 succeeded before wave 2 started

# 3. The running digest matches the committed one
kubectl -n geoportal get deploy geonode -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
grep newTag clusters/production/geoportal/kustomization.yaml

# 4. A revert actually restores the previous state
git revert --no-edit HEAD && git push
argocd app wait geoportal-production --health --timeout 300
#   expect: healthy, on the previous digest

# 5. Nothing drifted after the sync
argocd app diff geoportal-production
#   expect: no differences

Troubleshooting matrix

Symptom Likely cause Fix
The application starts before the migration finishes No sync waves, or the migration is not a pre-sync hook Add waves; make the migration a hook with backoffLimit: 0
A failed migration leaves the schema half-applied Migration retried automatically Disable retries on the migration job and require a human decision
Rollback restores the image but not the behaviour The migration was not backward compatible Split destructive changes into add-then-backfill-then-remove releases
A manual fix is reverted within a minute Automated sync is still enforcing the repository Suspend sync before manual intervention, and reconcile afterwards
Production drifts silently from the repository Self-heal disabled and nobody watches the diff Alert on the application’s sync status, not only on its health
A promotion moved more than intended The overlay carried other uncommitted changes Promote only the digest field; keep configuration changes in their own commits
Sync succeeds but the portal is broken Health checks pass on liveness rather than on serving a real request Add a custom health check that renders a tile before the wave is called healthy

FAQ

Should production sync automatically?

Not for a portal with a database migration in the path. Automatic sync is excellent for stateless services and removes a useful pause for anything that changes a schema. Keep staging automatic so the feedback loop stays short, and make production a deliberate action with the diff in front of somebody.

How should database migrations be handled across a rollback?

By making them backward compatible, so that the previous application version runs against the new schema. That means adding before removing, backfilling separately, and deferring destructive changes to a later release once the rollback window has passed. The alternative — rolling the schema back too — is a restore, not a rollback.

What about the tile cache during a release?

If the release changes cartography, the cache is stale the moment it lands. Treat invalidation as part of the release rather than as a follow-up: a sync wave after the application that purges the affected layers, or a version in the tile path so old and new coexist and the old expires naturally.

Can the same repository serve several clusters?

Yes, and it should, with one path per cluster and one Argo CD application per path. What to avoid is a single application templated across clusters with the differences in a values file — it makes a promotion an edit to a shared file, which is exactly the change that is easiest to get wrong under pressure.

How does this interact with drift detection?

They complement each other. The reconciler enforces the manifests it manages; the drift detection described in detecting Terraform drift in GeoNode infrastructure covers the infrastructure beneath them, which the reconciler does not see. A portal needs both, and the boundary between them should be written down.

Should the migration job be part of the same application?

Yes, as a hook rather than as a separate application. Splitting it out makes the ordering somebody’s responsibility rather than the tool’s, and the ordering is the entire reason the hook exists.

What belongs in the health check that gates a wave?

Something that proves the service is doing its job, not that its process is running. For a portal that means rendering a known tile and running a catalogue query; for the renderer it means a real GetMap against a small extent. A liveness probe on a port lets the next wave start against a component that has not finished loading its styles, which is precisely the ordering the waves exist to enforce.

Up one level: Helm vs Kustomize for GeoNode Deployments.