Geospatial Portal

Restoring a GeoNode Portal from Cold Backup

This guide executes a full portal recovery into an empty environment: what to restore, in what order, and what to check before each subsequent step. It belongs to Backup & Disaster Recovery for Spatial Platforms, within the Infrastructure Orchestration & Configuration Management framework.

Prerequisites

  • A working database backup with a known restore point, per taking consistent PostGIS backups with pgBackRest.
  • Versioned object storage holding source files, and a backup of the rendering engine’s data directory.
  • The manifests in version control, and the ability to deploy them into an empty namespace.
  • Credentials for the new environment, and the backup encryption passphrase, both obtainable without the environment being recovered.

Restore from manifests, not from machine images

The instinct under pressure is to restore everything, including the stateless tier, from whatever copy exists. Resist it. Rebuilding the stateless tier from version control is what proves that the declared state is complete, and it is the only way the recovery finds the configuration that exists only because someone applied it by hand two years ago.

Restore images, or rebuild from the repository Two approaches compared by what they reproduce, what they hide, and what each teaches about the completeness of the declared state. restore machine images faster, and reproduces the drift too hides what the repository is missing carries whatever was on the node the gap is discovered later, when the images are no longer available rebuild from manifests restores only data and engine config exposes undeclared configuration now the environment ends up reproducible costs more during the drill, and the drill is where you want to find out

Step-by-step implementation

1. Restore the spatial database and stop 

# Recover to the chosen point. Everything downstream depends on this being right,
# so verify before moving on rather than after.
pgbackrest --stanza=geoportal --type=time \
  --target="2026-08-09 06:00:00+00" --target-action=promote restore

pg_ctl -D /var/lib/postgresql/16/main start

psql -d geoportal -Atc "
  SELECT 'parcels', count(*) FROM parcels
  UNION ALL SELECT 'layers', count(*) FROM base_resourcebase;"
#   compare against the recorded production figures for that timestamp

2. Restore the object store 

# Restore to the same point in time using object versions, not the current state.
aws s3api list-object-versions --bucket agency-geonode-media \
  --query "Versions[?LastModified<='2026-08-09T06:00:00Z']" > versions.json
python3 restore_versions.py versions.json s3://agency-geonode-media-restored/

# Then check the count and total size against what the database expects.
psql -d geoportal -Atc "SELECT count(*) FROM base_link WHERE link_type='original';"
aws s3 ls --recursive --summarize s3://agency-geonode-media-restored/ | tail -2

3. Restore the rendering engine’s configuration

This is the step most often missing from a plan, and without it the portal starts with no workspaces, no styles, and no datastores.

# The engine's data directory is a first-class artefact, not a derived one.
tar -xzf geoserver-data-2026-08-09.tar.gz -C /opt/geoserver/

# Its datastore definitions embed credentials that no longer apply in the new
# environment; re-point them through the REST API rather than editing files.
for STORE in $(curl -sf -u "$GS_ADMIN" \
    "http://localhost:8080/geoserver/rest/workspaces/geoportal/datastores.json" \
    | jq -r '.dataStores.dataStore[].name'); do
  curl -sf -u "$GS_ADMIN" -XPUT -H "Content-Type: application/json" \
    -d "{\"dataStore\":{\"connectionParameters\":{\"entry\":[
          {\"@key\":\"host\",\"\$\":\"postgis.geoportal.svc\"},
          {\"@key\":\"passwd\",\"\$\":\"${NEW_DB_PASSWORD}\"}]}}}" \
    "http://localhost:8080/geoserver/rest/workspaces/geoportal/datastores/${STORE}.json"
done

4. Bring the portal up read-only 

# Deploy from the repository, with writes disabled until the data is verified.
# A recovery that accepts edits before anyone has checked what was restored
# turns a second restore attempt into a data-merge problem.
env:
  - name: READ_ONLY_MODE
    value: "true"
  - name: DISABLE_UPLOADS
    value: "true"
kubectl -n geoportal rollout status deploy/geonode --timeout=300s
curl -sf -o /dev/null -w '%{http_code}\n' \
  "https://portal.example.gov/geoserver/wms?service=WMS&request=GetMap&layers=parcels&bbox=-1.31,50.90,-1.28,50.92&width=256&height=256&format=image/png&srs=EPSG:4326"
#   expect: 200, and a tile that actually contains features

5. Rebuild derived state, then resume workers 

# The search index is derived; rebuild it only once the catalogue is verified.
python3 manage.py rebuild_search_index --noinput
curl -s "http://elasticsearch:9200/catalogue/_count" | jq .count
#   expect: matching the catalogue record count

# Workers last, and only after the queue has been reconciled rather than replayed.
kubectl -n geoportal scale deploy/celery-worker --replicas=4
The check that gates each step Five gates in sequence, each naming the assertion that must pass before the next step begins. after the database row counts and spatial extents match the recorded figures for the target time after the object store object count matches the database's link records; no zero-byte objects after the engine config a capabilities request lists every expected layer, not a subset after the application a tile renders WITH features — a blank tile is a failure, not a pass after the index indexed record count matches the catalogue; a sample search returns results

6. Reopen writes deliberately

Turning writes back on is a decision with an owner, not the last line of a script. Announce the restore point so that anyone whose work falls after it knows to redo it, then remove the read-only flag.

Three things that precede reopening writes Acceptance of the data loss, notification of affected editors, and an audit record, before the read-only flag is removed. 1 · acceptance the data owner accepts the loss the point implies not a platform decision 2 · notification editors are told what window is missing specific, not "some data" 3 · record restore point and decision in the audit trail answers next year's question Only then remove the read-only flag. Edits made on top of an unverified restore cannot be separated from it afterwards, so a decision to restore to a different point stops being a repeat of this procedure and becomes a data-merge project.

What to record while the recovery is running

A recovery is also the moment when the most useful information about the platform is available and least likely to be written down. Three things are worth capturing as it happens, because they cannot be reconstructed afterwards and they determine how the next one goes.

The first is timings, per step, from one clock. Not an overall duration — the per-step breakdown is what tells you whether the objective is missed because of the database restore, the index rebuild, or forty minutes spent looking for a credential. Teams consistently guess this wrong before measuring it; the step that dominates is rarely the one expected.

The second is every command that was typed and is not in the runbook. Under pressure these feel like trivial improvisations — a flag added, a service restarted, a permission granted — and each one is a gap in the written procedure that will be hit again by somebody less familiar. Capturing them costs nothing at the time and is nearly impossible to recall a day later.

The third is the state readers were in throughout: when the portal started answering at all, when it started serving correct tiles, and when search worked. That timeline is what an incident report needs, and it is materially different from the timeline of what the platform team was doing. A portal that was serving a degraded basemap for two of the three hours had a very different impact from one that was returning errors for the same period, and only somebody watching from the outside will know which it was.

Assign the recording to somebody who is not performing the recovery. It is not a secondary task that the person restoring can pick up between commands, and a recovery with no scribe reliably produces a report written from memory the following week — which is to say, a report that omits exactly the details that would have improved the next one.

Troubleshooting matrix

Symptom Likely cause Fix
Layers list is empty after a successful database restore The engine’s data directory was not restored Restore it; it is not derived state and cannot be reconstructed
Layers exist but every request returns an error Datastores still point at the old host or credentials Re-point through the REST API, then reset the connection pools
Thumbnails and downloads are broken Object store restored to current state rather than to the target time Restore by object version at the same point in time as the database
Search returns nothing Index rebuilt before the catalogue restore finished Gate the rebuild on the database check, and re-run it
Duplicate published layers after workers resume The queue was restored and replayed Reconcile instead of replaying; rely on idempotent tasks
Tiles render blank in the recovered environment Styles restored but fonts or sprites missing from the image Rebuild the renderer from its image; confirm assets are in the image, not the volume
Recovery takes far longer than the objective Serial restore and no measured baseline Parallelise, and measure the real duration during a drill rather than estimating

FAQ

How long should a full recovery take?

Whatever the drill measured, which is the only number worth quoting. For a mid-sized portal with a few hundred gigabytes of spatial data, a parallel restore and a rebuilt stateless tier typically lands in the low hours; the parts that surprise teams are the index rebuild and the tile cache warm, both of which are derived and both of which can be deferred behind a degraded mode.

Can the portal serve while the tile cache is still cold?

Yes, and it will be slow. This is exactly the window that a pre-built fallback archive covers, as described in serving a degraded basemap during an outage — announce the degraded state, serve the archive, and warm the real cache in the background.

What if the restore point predates a schema migration?

Restore the database and then apply the migrations in order, exactly as a deployment would. The failure to avoid is restoring old data into a new application version and assuming the schema will be reconciled implicitly; check the migration state explicitly before starting the application.

Should the recovered environment keep the same hostnames?

Eventually yes, and not immediately. Bring it up on a separate name for verification, so that a partly-verified portal is not being consumed by real clients, and cut the public name over as a deliberate step once the checks have passed.

Should the recovered environment be kept after the exercise or incident?

Only if it is now production. Otherwise destroy it, promptly and verifiably — it holds a full copy of production data in an environment with weaker controls, less monitoring, and credentials that were created quickly. A leftover recovery environment is one of the more common ways a portal’s data ends up somewhere nobody is watching, and it is entirely avoidable with a teardown step in the same runbook.

Up one level: Backup & Disaster Recovery for Spatial Platforms.