Geospatial Portal

Rotating GeoServer Credentials Without Downtime

A zero-downtime procedure for replacing the GeoServer admin password, REST API user, and PostGIS datastore secrets while keeping every WMS and WFS request served.

This runbook is a hands-on companion to Security Boundary Mapping for OGC Services and sits within the wider Core Portal Architecture & Security Boundaries practice; read the parent guide first if you need the reasoning behind which trust boundaries each credential crosses. Here the scope is deliberately narrow: how to overlap old and new secrets so that no in-flight OGC request ever hits an authentication failure during the swap.

Prerequisites

Credential rotation fails most often because one holder of the secret is forgotten. Confirm all of the following before you touch anything.

  • GeoServer 2.24+ running behind a load balancer or Kubernetes Service, with at least two replicas so a rolling restart never drops the endpoint to zero.
  • Admin access to the GeoServer web UI or REST API, plus kubectl write access to the namespace holding the Secret (examples use -n geoportal).
  • PostgreSQL superuser or role-owner access to the PostGIS backend so you can create and later drop a database role.
  • A password manager or Vault path to stage the new secret; never paste it into shell history — read it from a file or read -s.
  • An inventory of every credential holder: the GeoServer admin account, the REST automation user, and each PostGIS datastore password embedded in a store’s datastore.xml.
  • A monitoring probe on /geoserver/web/ and one authenticated WFS GetFeature call so you can watch for a spike in 401/403 responses throughout the change.

The state machine below is the mental model for the whole change: every credential moves through prepare, then a dual-valid window where both the old and new secret authenticate, then cutover, and only then revocation.

Zero-downtime credential rotation state machine Four states are laid out left to right. Prepare mints the new secret while the old one still serves all traffic. Dual-valid is the safety window in which both the old and new credential authenticate successfully, so no request can fail while configuration propagates. Cutover switches every client and datastore to the new secret only. Revoke destroys the old credential and confirms no authentication failures follow. A dashed rollback arrow returns from cutover to dual-valid if error rates rise. Prepare mint new secret old still serves Dual-valid both authenticate safety window Cutover new secret only rolling restart Revoke drop old role verify no 401s rollback if error rate rises

Step-by-step implementation

1. Inventory every holder of the credential

Rotation is only safe once you know every place the secret lives. GeoServer keeps datastore passwords inside each store’s datastore.xml under its data directory, the admin and REST users live in the security subsystem, and the Kubernetes Secret holds whatever the pod reads at boot. Enumerate them first.

# List every datastore config that embeds a PostGIS password
grep -rl "<entry key=\"passwd\">" \
  "$GEOSERVER_DATA_DIR"/workspaces/*/*/datastore.xml

# Show the security roles and users GeoServer knows about
curl -su "$GS_ADMIN" \
  https://portal.example.org/geoserver/rest/security/usergroup/users.json

# Confirm which keys the running Secret exposes to the pod
kubectl get secret geoserver-credentials -n geoportal \
  -o jsonpath='{.data}' | tr ',' '\n'

Record the workspace and store name for each PostGIS datastore. A single GeoServer often fronts several stores that share one database role, so decide up front whether you are rotating one shared role or one role per store.

2. Mint the new secret and create a parallel PostGIS role

Generate the new password out of band, then create a new PostGIS role rather than changing the existing one in place. A distinct role is what makes the dual-valid window possible: both roles can read the same tables at once.

-- Run as a role owner on the PostGIS backend
CREATE ROLE geoserver_app_v2 LOGIN PASSWORD 'REDACTED_STRONG_SECRET';

-- Grant it exactly what the old role had, no more
GRANT geoserver_app_ro TO geoserver_app_v2;
GRANT USAGE ON SCHEMA public TO geoserver_app_v2;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO geoserver_app_v2;

Granting through an existing group role such as geoserver_app_ro keeps privileges identical to the outgoing account and avoids a subtle authorization drift where the new role can see more (or less) than the old one. Keep the connection ceiling in mind while both roles are live — two roles briefly double the session count, so review Optimizing PostgreSQL/PostGIS Connection Limits before you begin.

3. Open the dual-valid window

Before removing anything, make the new secret work everywhere the old one does. For the PostGIS layer both roles now authenticate, so any datastore can be flipped independently. For the GeoServer admin and REST users, add the new credential alongside the old rather than overwriting it, so automation that still holds the old password keeps succeeding.

# Add the new REST/automation user without deleting the old one
curl -su "$GS_ADMIN" -XPOST \
  -H "Content-Type: application/json" \
  -d '{"user":{"userName":"rest_ci_v2","password":"REDACTED","enabled":true}}' \
  https://portal.example.org/geoserver/rest/security/usergroup/users

# Grant the new automation user the same role as the old one
curl -su "$GS_ADMIN" -XPOST \
  https://portal.example.org/geoserver/rest/security/roles/role/ADMIN/user/rest_ci_v2

At this point nothing has been taken away. Both PostGIS roles authenticate, both REST users authenticate, and the admin can still log in with the old password. Confirm the window is genuinely open before proceeding — a failed grant here is far cheaper to fix than a half-cutover portal.

4. Push the new datastore password via the REST API

Update each PostGIS datastore to the new role through the REST API rather than editing datastore.xml by hand; the API re-encrypts the password with GeoServer’s configured encoder and reloads the store’s connection pool without a restart. Send only the changed connection parameters.

WS=geoportal
STORE=parcels_pg
curl -su "$GS_ADMIN" -XPUT \
  -H "Content-Type: application/json" \
  -d '{"dataStore":{"connectionParameters":{"entry":[
        {"@key":"user","$":"geoserver_app_v2"},
        {"@key":"passwd","$":"REDACTED_STRONG_SECRET"}
      ]}}}' \
  "https://portal.example.org/geoserver/rest/workspaces/$WS/datastores/$STORE.json"

# Force the store to re-pool connections with the new role
curl -su "$GS_ADMIN" -XPOST \
  "https://portal.example.org/geoserver/rest/reset"

Because the old PostGIS role is still valid, any request that was mid-flight against the old connection pool completes normally while new connections open under geoserver_app_v2. Repeat for every datastore in your inventory.

5. Roll the Kubernetes Secret and restart

Now update the Secret that seeds the admin and REST passwords at boot, then trigger a rolling restart so each replica picks up the new values one at a time. Because you have more than one replica and both credentials are still valid, the endpoint stays up throughout.

# Replace the Secret in place (values are base64 on write)
kubectl create secret generic geoserver-credentials -n geoportal \
  --from-literal=admin-password='REDACTED' \
  --from-literal=rest-password='REDACTED' \
  --dry-run=client -o yaml | kubectl apply -f -

# Roll one pod at a time; readiness gating keeps the Service serving
kubectl rollout restart statefulset/geoserver -n geoportal
kubectl rollout status  statefulset/geoserver -n geoportal --timeout=180s

A StatefulSet with maxUnavailable effectively one and a readiness probe on /geoserver/web/ ensures the load balancer only routes to pods that have already booted with the new secret. If your MapProxy tier authenticates to GeoServer, coordinate this restart with its config reload — the pattern is covered in Securing MapProxy with Nginx and ModSecurity.

6. Revoke the old credential and confirm

Only after every holder authenticates with the new secret do you close the window. Delete the old REST user, disable the old admin password path, and drop the retired PostGIS role.

# Remove the superseded automation user
curl -su "$GS_ADMIN" -XDELETE \
  https://portal.example.org/geoserver/rest/security/usergroup/user/rest_ci

# Drop the old role once no session uses it
psql -h db -U postgres -d geoportal -c \
  "SELECT count(*) FROM pg_stat_activity WHERE usename='geoserver_app';"
# expect 0 before the next line
psql -h db -U postgres -d geoportal -c "DROP ROLE geoserver_app;"

If pg_stat_activity still shows sessions for the old role, a datastore was missed in step 4 — do not drop the role until that count reaches zero, or those requests will start returning 500.

Verification

Confirm the new secret is fully in force and the old one is genuinely dead.

# 1. An authenticated WFS request still succeeds end to end
curl -so /dev/null -w '%{http_code}\n' -u "$GS_APP_USER" \
  "https://portal.example.org/geoserver/wfs?service=WFS&version=2.0.0&request=GetCapabilities"
#   expect: 200

# 2. The old PostGIS role can no longer log in
PGPASSWORD='OLD_SECRET' psql -h db -U geoserver_app -d geoportal -c '\q'
#   expect: FATAL: role "geoserver_app" does not exist

# 3. No authentication failures in the last 5 minutes of GeoServer logs
kubectl logs statefulset/geoserver -n geoportal --since=5m | grep -c "AuthenticationException"
#   expect: 0

# 4. Every datastore reports a healthy connection pool
curl -su "$GS_ADMIN" \
  "https://portal.example.org/geoserver/rest/workspaces/geoportal/datastores.json" \
  | grep -o '"name":"[^"]*"'

A 200 on the WFS call, a rejected login for the old role, and a zero count of authentication exceptions together prove the rotation completed without a gap. Fold checks 1 and 3 into your monitoring so the next rotation is observed automatically rather than watched by hand.

Troubleshooting matrix

Symptom Likely cause Fix
WFS/WMS requests return 500 right after dropping the old role A datastore still pointed at the old PostGIS role when it was dropped Recreate the role, PUT the new credential to the missed store (step 4), then drop again once pg_stat_activity is zero
REST automation starts failing with 401 mid-rotation Old REST user deleted before the new one was granted its role Recreate the user and re-grant the role; never delete before the dual-valid window is confirmed
Admin login rejected after the rolling restart Secret key name does not match what the pod reads at boot Diff kubectl get secret ... -o yaml against the container’s env mapping; fix the key and re-roll
datastore.xml still shows the old password after a PUT Edited the file directly instead of via REST, so the encoder never re-ran Revert the manual edit and PUT through /rest/...; run POST /rest/reset to re-pool
Connection count alarms fire during the window Both roles hold sessions simultaneously, exceeding the pool ceiling Shorten the dual-valid window, or temporarily raise the pooler limit per the connection-limits guide
New role authenticates but returns no rows Group-role grant omitted, so privileges differ from the old role GRANT the same group role and table SELECT the old account had, then retry

For which of these secrets crosses an external trust boundary versus an internal one — and therefore how urgently each must rotate — consult the parent Security Boundary Mapping for OGC Services guide.

Up one level: Security Boundary Mapping for OGC Services.