Geospatial Portal

Running a Disaster Recovery Game Day

This guide runs a recovery exercise that produces evidence rather than reassurance: a stated hypothesis, a scoped scenario, measured timings, and a findings list that turns into scheduled work. It belongs to Backup & Disaster Recovery for Spatial Platforms, within the Infrastructure Orchestration & Configuration Management framework.

Prerequisites

  • A documented recovery sequence, as in restoring a GeoNode portal from cold backup.
  • An isolated environment to recover into, with its own credentials and no path to production.
  • A recorded set of production figures — row counts, extents, layer counts — for the target restore point.
  • A stated recovery time objective, so the exercise has a number to pass or fail against.

A game day is an experiment, not a rehearsal

The difference matters. A rehearsal practises a known procedure and succeeds by definition if everyone follows it. An experiment states a hypothesis in advance and can fail, which is the only way it produces information.

Before, during and after a recovery experiment Three phases with what must be fixed in advance, what is recorded during, and what the output is afterwards. before the hypothesis, written down the scenario and its scope success criteria and a time target fixed in advance, not adjusted later during follow only the written runbook record every deviation, timed a scribe who is not restoring improvising is a finding, not a fix after measured timings vs the target findings with owners and dates runbook updated the same week an exercise with no findings An exercise that produces no findings usually means the scenario was too gentle or the runbook was written during the exercise. Neither is a reason for confidence.

Step-by-step implementation

1. Write the hypothesis before touching anything 

# DR exercise — 2026-08-09

Scenario: the production database and rendering engine are unavailable and cannot
be recovered in place. Object storage and backups are intact.

Hypothesis:
  We can bring a read-only portal serving the parcels and basemap layers, from
  cold backup, into an empty namespace, within 3 hours, losing no more than
  15 minutes of edits.

Success criteria:
  1. GetCapabilities lists all 41 expected layers.
  2. A parcels tile renders WITH features at z15 over the city centre.
  3. Row count and extent for parcels match the recorded figures at the target time.
  4. Catalogue search returns results for a known query.
  5. Elapsed wall-clock time from start to criterion 4 is under 3 hours.

Out of scope: write traffic, the search index rebuild beyond criterion 4,
tile cache warming.

2. Choose a scenario that is likely, not dramatic

The instinct is to simulate a region loss. The more informative exercise is usually the mundane one, because it is the one that will actually happen.

Which scenario to exercise first Four scenarios ordered by likelihood, each with what the exercise teaches and roughly how long it takes to run. SCENARIO TEACHES LIKELIHOOD a bad publish the single-dataset restore path, and the version swap very likely database lost the core sequence and the real restore duration likely cluster lost whether the manifests are actually complete occasional region lost the cross-region path, and the write-posture decision rare

3. Run it with a scribe and a stopwatch 

# Everything is timed from a single clock, and every step's duration recorded.
exec > >(ts '[%Y-%m-%d %H:%M:%S]' | tee -a dr-exercise-2026-08-09.log) 2>&1

echo "STEP 1 restore database"
time pgbackrest --stanza=geoportal --type=time \
  --target="2026-08-09 06:00:00+00" --pg1-path=/var/lib/postgresql/dr restore

echo "STEP 2 verify data"
time psql -h /tmp -p 5433 -d geoportal -Atc \
  "SELECT count(*) FROM parcels; SELECT ST_Extent(geom)::text FROM parcels;"

The scribe’s job is to record what was actually done, not what the runbook says. Every command typed that is not in the runbook is a finding, and it is much easier to notice at the time than to reconstruct afterwards.

4. Convert deviations into work, the same week

The exercise’s output is a findings list, and its value decays quickly. A finding recorded and scheduled within the week becomes a fix; the same finding three weeks later becomes a note in a document nobody reads before the next exercise.

Four kinds of finding, four kinds of follow-up Missing steps, missing artefacts, missed timings and missing owners, each with the type of work it becomes. a missing step somebody knew it and it was not written down edit the runbook now a missing artefact something was not backed up at all change the backup a missed timing the objective was not met engineer, or revise it a missing owner nobody could authorise a decision an organisational fix

The third row deserves an explicit choice rather than a default. If the recovery takes five hours against a three-hour objective, either the platform changes or the objective does — and stating which, with a date, is what stops the same finding recurring at every exercise for three years.

Verification 

# 1. Every success criterion has a recorded pass or fail
grep -E '^CRITERION [1-5]' dr-exercise-2026-08-09.log
#   expect: five lines, each PASS or FAIL — no blanks

# 2. The elapsed time is measured, not estimated
head -1 dr-exercise-2026-08-09.log; tail -1 dr-exercise-2026-08-09.log
#   expect: two timestamps, and their difference recorded in the report

# 3. The recovered portal serves real data, not an empty shell
curl -s "https://dr.portal.internal/geoserver/wms?service=WMS&request=GetCapabilities" \
  | grep -c '<Layer'
#   expect: 41, matching the recorded production figure

# 4. Every deviation from the runbook is captured
grep -c '^DEVIATION' dr-exercise-2026-08-09.log
#   expect: a number, and each one appearing in the findings list

# 5. The environment is destroyed afterwards
kubectl delete namespace geoportal-dr
aws s3 rm --recursive s3://agency-geonode-media-restored/
#   expect: no orphaned copy of production data left behind

Check 5 is easy to skip and worth automating. A recovery exercise produces a full copy of production data in a less-protected environment, and an environment left running after the exercise is a data exposure that nobody is monitoring.

Keeping the exercise honest as the platform changes

The most common way a game-day programme decays is not that exercises stop, but that they become easier without anyone deciding to make them easier. The scenario gets narrower, the environment gets more prepared in advance, the runbook gets read the day before, and after a year the exercise confirms only that a well-rehearsed team can follow a familiar script.

Three counters help. Rotate the scenario, so the same path is not exercised every time and the less-travelled ones — the single-dataset restore, the engine configuration, the credential retrieval — get their turn. Rotate the operator, so the runbook is regularly read by somebody who did not write it and does not remember the unwritten parts. And keep at least one element unannounced: not the date, which people need in order to attend, but one detail of the scenario, so that some of the response is genuinely improvised.

Track two numbers across exercises rather than judging each one alone. The measured recovery duration should trend downward or stay flat as the platform grows; a rising number is a signal that the recovery path has accumulated steps faster than it has been improved. And the count of findings should not trend to zero — an exercise that reliably produces nothing is measuring the team’s familiarity rather than the platform’s recoverability, and it is time to make the scenario harder.

Finally, make the results visible outside the platform team. The stakeholders who own the data are the ones who will be asked to accept data loss during a real event, and a quarterly one-page report showing measured timings and outstanding findings is what turns that conversation from a surprise into a continuation of one they have already been having.

Troubleshooting matrix

Symptom Likely cause Fix
The exercise “passed” but taught nothing The runbook was written during the exercise by whoever knew Fix the criteria in advance; treat improvisation as a finding
Findings from the last exercise recur No owner or date attached, so nothing was scheduled Convert each finding into a dated task in the same week
The recovery could not start Backup passphrase or credentials only available in production Store recovery credentials outside the environment being recovered
Timings vary wildly between exercises Different data volumes or contended infrastructure Record the data size with the timing; compare like with like
Nobody could decide whether to accept data loss The decision has no named owner Name the owner in the plan, not during the incident
The exercise disrupted production The DR environment reached production credentials or data Isolate the environment fully; verify the isolation before starting
The report is never read It is long, and written for nobody in particular One page: hypothesis, result, timings, findings with owners

FAQ

How often should a game day run?

Quarterly for the core sequence, and after any change that touches the recovery path — a database version upgrade, a change of storage provider, a new component in the stateful tier. The single-dataset restore is cheap enough to exercise monthly, and it is the one that matches the most likely incident.

Should the exercise run against production?

The restore should not, and the failover might, once the sequence is well understood and the write posture is decided. Start in an isolated environment. Exercising a real failover in production is a legitimate ambition, and it belongs several successful exercises later, announced, and on a quiet afternoon.

Who should perform the recovery?

Not the person who wrote the runbook, and ideally somebody who could plausibly be on call at three in the morning. A runbook that only its author can follow is a document about a capability rather than the capability itself, and the fastest way to discover that is to hand it to somebody else.

What if the exercise fails badly?

That is the exercise working. A failed exercise on a Tuesday afternoon, with everyone available and no real data at risk, is the cheapest possible way to learn what is missing. The only bad outcome is a failure that produces no findings — which usually means nobody was recording.

Up one level: Backup & Disaster Recovery for Spatial Platforms.