Geospatial Portal

Detecting Terraform Drift in GeoNode Infrastructure

How to catch out-of-band changes to GeoNode’s cloud infrastructure by running a scheduled terraform plan in CI, comparing state to reality, and alerting the moment they diverge.

This page is a hands-on companion to Environment Parity in Geospatial CI Pipelines within the broader Infrastructure Orchestration & Configuration Management practice; read the parent guide for how environments are kept identical, then use this procedure to prove they stay that way. Drift — a security group edited in the console, an RDS parameter tuned by hand, a load balancer someone deleted — silently breaks the guarantee that your code describes your running GeoNode stack. Scheduled drift detection turns that silent divergence into a loud, actionable signal.

Prerequisites

Have these ready before wiring the pipeline, because each is a precondition for a reliable drift signal.

  • Terraform 1.6+ (or OpenTofu 1.6+) with a remote backend — S3 + DynamoDB lock, Terraform Cloud, or GCS — holding the GeoNode stack’s state.
  • A CI runner with read-only cloud credentials scoped to plan; drift detection must never be able to mutate infrastructure.
  • The GeoNode Terraform configuration in version control, applied at least once so state reflects a known-good baseline.
  • Optionally driftctl 0.40+ to catch resources that exist in the cloud but are absent from state (out-of-band creations Terraform cannot see).
  • A notification sink (Slack webhook, PagerDuty, or an issue-tracker token) and permission for CI to open issues or post alerts.

Step-by-step implementation

The core trick is terraform plan -detailed-exitcode, whose exit code is the drift signal: 0 means no changes, 2 means the plan found differences, and 1 means the command errored. A scheduled job runs the plan against locked state, interprets the exit code, and routes a report. The diagram shows that loop.

Scheduled drift-detection loop comparing Terraform state to live GeoNode infrastructure A scheduled trigger, such as an hourly cron, starts a CI job. The job reads the locked remote state and runs terraform plan with the detailed exit code flag using read-only credentials, comparing the recorded state against the real GeoNode infrastructure of RDS, security groups, and load balancers. The command's exit code drives a three-way branch. Exit code zero means state and reality agree, and the job ends green with no drift. Exit code two means the plan detected differences: the job publishes a drift report, alerts the team, and optionally opens an issue or blocks promotion. Exit code one means the command itself errored, which is escalated separately as a pipeline failure. A dashed return arrow shows the loop repeating on the next schedule. schedule hourly cron terraform plan -detailed-exitcode read-only creds locked remote state live infrastructure RDS · SGs · load balancers exit code 0 · 1 · 2 0 · in sync green, no drift 2 · DRIFT report · alert · issue 1 · error escalate pipeline repeats on the next schedule

1. Run a read-only plan with a detailed exit code

The whole detection hinges on one flag. -detailed-exitcode promotes the “changes present” case to exit 2, distinct from a genuine error (1). Run the plan with -refresh-only semantics against locked state, using credentials that cannot apply.

terraform init -input=false -backend-config=backend.hcl

set +e
terraform plan -detailed-exitcode -lock=true -input=false \
  -refresh=true -out=drift.tfplan
code=$?
set -e

case "$code" in
  0) echo "No drift: state matches infrastructure" ;;
  2) echo "DRIFT DETECTED: plan found differences" ;;
  1) echo "ERROR: plan failed" >&2; exit 1 ;;
esac

The -refresh=true step is what actually reads live resource attributes and reconciles them against state, so a hand-edited security group surfaces as a proposed change even though no code changed.

2. Store and compare state artifacts

Capture the machine-readable plan on every run so a human can see what drifted, not just that something did. Convert the binary plan to JSON and keep it as a build artifact keyed by timestamp.

terraform show -json drift.tfplan > drift.json

# Count resources with a non-no-op change (create/update/delete/replace)
drift_count=$(jq '[.resource_changes[]
  | select(.change.actions != ["no-op"])] | length' drift.json)
echo "drifted_resources=$drift_count" >> "$GITHUB_OUTPUT"

Comparing successive drift.json artifacts turns a single alert into a trend — you can see whether the same resource drifts repeatedly, which usually points at automation fighting Terraform.

3. Schedule the check in CI

Run the detection on a cron trigger, not only on pushes, so drift introduced between deploys is caught within the hour. The workflow below is valid GitHub Actions YAML; the schedule and the exit-code interpretation are the load-bearing parts.

name: terraform-drift-detection
on:
  schedule:
    - cron: "0 * * * *"        # hourly drift sweep
  workflow_dispatch: {}

permissions:
  contents: read
  issues: write                 # allow opening a drift issue

jobs:
  drift:
    runs-on: ubuntu-latest
    env:
      AWS_REGION: eu-west-1
      TF_IN_AUTOMATION: "true"
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "1.9.5"
      - name: Init
        run: terraform init -input=false -backend-config=backend.hcl
      - name: Plan with detailed exit code
        id: plan
        run: |
          set +e
          terraform plan -detailed-exitcode -lock=true -input=false -out=drift.tfplan
          echo "exitcode=$?" >> "$GITHUB_OUTPUT"
      - name: Report drift
        if: steps.plan.outputs.exitcode == '2'
        run: |
          terraform show -no-color drift.tfplan | tee drift-report.txt
          echo "::warning::Terraform drift detected in GeoNode infrastructure"
      - name: Fail on drift
        if: steps.plan.outputs.exitcode == '2'
        run: exit 1
      - name: Fail on error
        if: steps.plan.outputs.exitcode == '1'
        run: exit 1

4. Catch out-of-band resources with driftctl

terraform plan only sees resources Terraform manages. A database or bucket created entirely outside Terraform is invisible to it. driftctl scans the cloud account and diffs it against state, surfacing unmanaged resources — the drift Terraform structurally cannot report.

driftctl scan \
  --from tfstate+s3://geonode-tfstate/prod/terraform.tfstate \
  --output json://driftctl-report.json \
  --output console://

# Non-zero exit if unmanaged or drifted resources exceed the tolerance

Feed the same guardrails you use for environment promotion; the parent workflow that applies this Terraform for GeoNode environments is described in Syncing GeoNode Environments with Terraform.

5. Publish a report, alert, and block promotion

Turn exit 2 into action: post the human-readable plan to chat, open an issue so the drift is tracked, and — for protected environments — make the drift job a required check so a promotion cannot proceed while infrastructure disagrees with code. The GeoNode module whose drift you are watching is typically packaged as HCL like this.

module "geonode" {
  source = "./modules/geonode"

  environment          = "prod"
  rds_instance_class   = "db.r6g.xlarge"
  rds_parameter_group  = aws_db_parameter_group.geonode.name
  alb_security_groups  = [aws_security_group.geonode_alb.id]
  desired_worker_count = 4

  # Lock drift-prone attributes so a console edit is flagged, not absorbed.
  lifecycle_ignore_tags = false
}

Whether this drift gate belongs alongside a Helm-based release or a Kustomize overlay depends on your packaging model, weighed in Helm vs Kustomize for GeoNode Deployments.

Not all drift deserves the same response

A drift detector that treats every difference as an incident trains its audience to ignore it, because most differences are benign: a cloud provider adds a default tag, an autoscaler changes a replica count that the configuration deliberately leaves unmanaged, a certificate is renewed by the mechanism designed to renew it. Sorting drift by cause before deciding what to do with it is what makes the report readable at all.

Sorting drift by cause Four bands from benign to serious: provider defaults, dynamic values, emergency changes, and unexplained changes, each with the response it warrants. Provider defaults tags, generated names, computed attributes adopt, or ignore explicitly nobody did this appears on every plan until handled Dynamic by design replica counts under an autoscaler, scaling state exclude from management, deliberately the system did this otherwise every apply fights the autoscaler Emergency change a firewall rule opened during an incident back-fill into configuration within days a person did this, knowingly legitimate, but temporary by definition Unexplained a change with no ticket, commit or incident behind it investigate — this is the alert

The value of the sorting is that the bottom band becomes rare enough to take seriously. A drift report that lists three hundred provider-authored differences hides a single unexplained security-group change perfectly; one that lists nothing but the unexplained change makes it impossible to miss.

Verification

Prove the detector reports 0 on a clean stack and 2 after a deliberate out-of-band change.

# 1. Baseline: a freshly applied stack reports no drift (exit 0)
terraform plan -detailed-exitcode -lock=true; echo "exit=$?"
#   No changes. Your infrastructure matches the configuration.
#   exit=0

# 2. Introduce drift out of band, e.g. edit a security group rule by hand
aws ec2 authorize-security-group-ingress \
  --group-id sg-0geonode --protocol tcp --port 9999 --cidr 0.0.0.0/0

# 3. The next plan must report drift (exit 2) and name the resource
terraform plan -detailed-exitcode -lock=true; echo "exit=$?"
#   ~ aws_security_group.geonode_alb will be updated in place
#   exit=2

# 4. driftctl also catches resources that were never in state
driftctl scan --from tfstate+s3://geonode-tfstate/prod/terraform.tfstate
#   Found N unmanaged resource(s)

An exit=0 on the clean baseline followed by exit=2 after the console edit confirms the pipeline reacts to exactly the changes it should.

Detecting drift without holding a lock all day

A scheduled drift check runs the same machinery as a deployment, which means it can take the state lock, and a lock held by a cron job at the moment of an incident is a genuinely bad afternoon. It also means a misconfigured job can apply rather than plan, which is worse.

Run drift detection with a read-only credential and a refresh-only plan, in a workspace that cannot apply, and give it a lock timeout short enough that it yields to a human immediately. If the check cannot obtain the lock, that is a successful outcome — somebody is deploying, and the drift question can wait for the next run.

Four guardrails for automated drift detection Four cards describing the credential, locking, execution and output constraints that keep a scheduled drift check from interfering with deployments. read-only credentials that cannot create, modify or delete a mistake in the job cannot change anything short lock wait give up in seconds if the state is busy a deploy always wins against a report plan only refresh and compare; no apply path exists self-healing is a separate, reviewed decision report out sorted by cause, not by resource the unexplained band is the only alert

Automatic remediation is tempting and should be a separate decision taken per resource type, not a global setting. Reverting an unexplained firewall change automatically is defensible; reverting an unexplained database parameter while an engineer is mid-incident is how a detector becomes the cause.

Troubleshooting matrix

Symptom Likely cause Fix
Plan always reports drift on every run Provider defaults or computed attributes differ from config Add lifecycle { ignore_changes = [...] } for genuinely computed fields; align defaults
Exit code always 0 despite real changes -detailed-exitcode omitted, so exit collapses to 0/1 Always pass -detailed-exitcode; treat 2 as drift
Scheduled job errors with state-lock timeout A concurrent apply holds the DynamoDB/backend lock Stagger the cron away from deploy windows; use -lock-timeout
Console-created resources never flagged terraform plan only sees managed resources Add driftctl (or import) to surface unmanaged resources
Drift job can mutate infrastructure Runner credentials scoped for apply, not read-only Issue plan-only credentials; deny apply in CI IAM policy
Noise from tag drift on every resource Cost-allocation tags injected out of band Normalize tags via default_tags; ignore the automated tag keys
Alert fires but no one can see what drifted Only the exit code is captured, not the plan Persist terraform show -json as an artifact and post the human-readable plan

For the exact exit-code contract and refresh semantics, the official Terraform plan command reference is authoritative.

Up one level: Environment Parity in Geospatial CI Pipelines.