Geospatial Portal

Storing GeoServer Secrets in Vault with Kubernetes Auth

This guide moves the rendering engine’s database credentials out of Kubernetes manifests and into Vault, authenticated by the pod’s own service-account identity rather than by a stored token. It belongs to Secrets Management for Geospatial Platforms, within the Core Portal Architecture & Security Boundaries framework.

Prerequisites

  • A Kubernetes cluster with a service-account token issuer reachable by Vault, and cluster-admin rights to create roles.
  • Vault 1.13 or newer, unsealed, with the Kubernetes auth method available to enable.
  • GeoServer deployed as a StatefulSet or Deployment with its data directory on a persistent volume.
  • A PostGIS instance whose application role you are willing to rotate as part of the migration, per rotating GeoServer credentials without downtime.

What the pod actually presents

The mechanism is worth understanding before configuring it, because the common failure is a role binding that is broader than intended. The pod presents its projected service-account token; Vault validates that token against the cluster’s public keys and reads the identity it asserts — namespace, service account name, and audience. The Vault role says which of those combinations may read which paths.

From a projected service-account token to a scoped secret read Four steps: present the projected token, verify it against cluster keys, match a role to a policy, and read only the permitted path. 1 · pod presents its projected service-account token 2 · Vault verifies signature against the cluster's public keys 3 · role to policy namespace + account must both match 4 · scoped read only the paths the policy names The two places this goes wrong Binding the role to a wildcard namespace or the default service account grants every workload in the cluster the same read. Omitting the audience check lets a token minted for another service be replayed against Vault. Both look like working configurations, because the intended workload still authenticates successfully.

Step-by-step implementation

1. Give the engine its own service account

Never bind Vault roles to the default service account. A dedicated account is what makes the grant specific, and it costs three lines.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: geoserver
  namespace: geoportal
automountServiceAccountToken: false   # the projected volume below is explicit

2. Enable Kubernetes auth in Vault and bind a narrow role 

vault auth enable -path=k8s-geoportal kubernetes

# Vault verifies tokens against the cluster's issuer and public keys.
vault write auth/k8s-geoportal/config \
    kubernetes_host="https://kubernetes.default.svc:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    disable_local_ca_jwt=false

# One role, one namespace, one service account, one audience. No wildcards.
vault write auth/k8s-geoportal/role/geoserver \
    bound_service_account_names=geoserver \
    bound_service_account_namespaces=geoportal \
    audience=vault \
    policies=geoserver-read \
    ttl=1h \
    max_ttl=4h

3. Write the policy at the narrowest path that works 

# geoserver-read.hcl — read one path, list nothing, write nothing.
path "kv/data/geoportal/geoserver/postgis" {
  capabilities = ["read"]
}

# Explicitly deny the parent so a future wildcard cannot widen this by accident.
path "kv/data/geoportal/*" {
  capabilities = ["deny"]
}

4. Project the token and inject the secret

The Vault agent injector turns the annotations below into an init container that authenticates, fetches, and writes the value to a file the engine reads at start-up.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: geoserver
  namespace: geoportal
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "geoserver"
        vault.hashicorp.com/auth-path: "auth/k8s-geoportal"
        vault.hashicorp.com/agent-inject-secret-postgis: "kv/data/geoportal/geoserver/postgis"
        # Render into the shape the engine expects, not raw JSON.
        vault.hashicorp.com/agent-inject-template-postgis: |
          {{- with secret "kv/data/geoportal/geoserver/postgis" -}}
          PGUSER={{ .Data.data.username }}
          PGPASSWORD={{ .Data.data.password }}
          {{- end }}
        vault.hashicorp.com/agent-inject-perms-postgis: "0400"
    spec:
      serviceAccountName: geoserver
      containers:
        - name: geoserver
          image: geoserver:2.24.2
          volumeMounts:
            - name: sa-token
              mountPath: /var/run/secrets/tokens
              readOnly: true
      volumes:
        - name: sa-token
          projected:
            sources:
              - serviceAccountToken:
                  path: vault-token
                  audience: vault          # must match the role's audience
                  expirationSeconds: 3600

5. Point the datastore at the injected value

GeoServer stores datastore passwords in its own configuration, so the migration is not complete until the datastore is updated through the REST API to use the credential Vault now owns — editing the file directly bypasses the encoder and leaves the store unusable.

# Read the injected credential inside the pod and push it via REST.
kubectl exec -n geoportal geoserver-0 -- sh -c '
  . /vault/secrets/postgis
  curl -sf -u "$GS_ADMIN" -XPUT -H "Content-Type: application/json" \
    -d "{\"dataStore\":{\"connectionParameters\":{\"entry\":[
          {\"@key\":\"user\",\"$\":\"$PGUSER\"},
          {\"@key\":\"passwd\",\"$\":\"$PGPASSWORD\"}]}}}" \
    "http://localhost:8080/geoserver/rest/workspaces/geoportal/datastores/postgis.json"
'

What breaks, and when

The migration has a specific failure profile: everything works until something restarts at an inconvenient moment. Knowing which dependencies are now in the start-up path is the difference between a controlled rollout and a surprise.

Three new start-up dependencies, and how each fails Identity issuer, Vault availability and policy grant, each with the failure symptom and the mitigation. DEPENDENCY SYMPTOM WHEN IT FAILS MITIGATION identity issuer cluster control plane pods stay in init, no clear error the injector retries silently alert on init-container duration Vault reachable and unsealed a restart cannot come back while running pods are unaffected cache last-good on the node policy still grants path unchanged 403 at fetch, pod never ready after an unrelated policy edit test policy changes in staging

The middle row is the one to plan for. A running pod holds its secret and is unaffected by Vault being unavailable; a restarting pod is not. That asymmetry means a Vault outage is invisible until the next deployment, node drain or crash — so the mitigation is to alert on the store’s health directly rather than to rely on the portal’s own symptoms.

Migrating without a window

The move from a manifest-held credential to a Vault-held one does not need an outage, because both can be valid at once for as long as you need. Run the two paths in parallel and cut over per component rather than all at once.

Cutting over one component at a time Four steps: seed Vault with the current value, migrate one component, migrate the rest, then delete the manifest secret and rotate. 1 · seed Vault with the current value, unchanged 2 · one component inject and confirm it reads the same value 3 · the rest workers, jobs, anything else holding a copy 4 · delete + rotate the rotation proves nothing was left behind Step 4 is the verification, not the cleanup Rotating immediately after the migration is the only reliable way to find a consumer that was still reading the manifest copy: it fails loudly, at a moment you chose, rather than silently at the next unplanned restart.

Do step four the same week, not the next quarter. A migration that stops after step three leaves both copies valid indefinitely, which is strictly worse than where it started: the same credential exists in the same places, plus a new one.

Verification 

# 1. The role authenticates only from the intended service account
kubectl -n geoportal exec deploy/other-app -- \
  vault write auth/k8s-geoportal/login role=geoserver \
  jwt=@/var/run/secrets/tokens/vault-token
#   expect: permission denied — the account name does not match

# 2. The engine's pod receives the rendered file, with tight permissions
kubectl -n geoportal exec geoserver-0 -- stat -c '%a %U %n' /vault/secrets/postgis
#   expect: 400 and the container's user

# 3. The credential is absent from the pod environment and from the manifest
kubectl -n geoportal exec geoserver-0 -- env | grep -c PGPASSWORD
#   expect: 0
kubectl -n geoportal get statefulset geoserver -o yaml | grep -ci password
#   expect: 0

# 4. The datastore actually works with the injected credential
curl -sf -o /dev/null -w '%{http_code}\n' \
  "https://portal.example.gov/geoserver/wfs?service=WFS&version=2.0.0&request=GetCapabilities"
#   expect: 200

# 5. Vault records the read, with the workload identity attached
vault read sys/internal/counters/activity | head
#   and: the audit log names auth/k8s-geoportal/geoserver, not a human

Troubleshooting matrix

Symptom Likely cause Fix
Pod stuck in init with no error in the app container The injector’s init container is retrying against an unreachable Vault Read the init container’s logs; alert on init duration, not just readiness
permission denied at login for the right service account Audience mismatch between the projected token and the Vault role Make audience identical in both places; it is not checked by name elsewhere
Secret file present but the engine still uses the old password The datastore configuration holds its own copy, updated only via REST Push the credential through the REST API and reset the connection pool
Every workload in the namespace can read the secret Role bound to the default service account Bind to a dedicated account and set automountServiceAccountToken: false
Credential rotates in Vault but pods keep the old one The value is read once at start-up Roll the deployment on rotation, or add a template with a restart hook
Vault outage takes down a scale-up but not running pods The fetch is in the start-up path with no cached fallback Cache last-good on the node and stagger fetches across pods
Audit shows reads from an unexpected role A second Vault role grants the same path Add the explicit deny on the parent path and re-audit the policy set

FAQ

Why not use a Kubernetes Secret and skip Vault entirely?

A cluster Secret is a genuine improvement over a value in a manifest, and for a small single-cluster portal it may be enough. What it lacks is rotation machinery, an access audit trail, and a trust boundary that survives cluster compromise — anyone who can read Secrets in the namespace can read the credential, permanently and without a record.

Does this remove the need to rotate the database password?

No, but it makes rotation cheap: the value exists in one place, so changing it is a single write plus a reload of the consumers. The overlap discipline is unchanged — create the new role, let both be valid, cut over, then revoke — and it is described in full in the rotation guide linked above.

Should the engine use dynamic database credentials instead of a static one?

It is attractive and it interacts badly with connection pooling: a pooled connection can outlive the lease that opened it, and the failure appears as intermittent authentication errors under load rather than at renewal. If you use dynamic credentials, set the pool’s maximum connection lifetime comfortably below the lease, and test it under sustained traffic rather than in a quiet staging environment.

What happens during a Vault upgrade or reseal?

Running pods are unaffected because they already hold their secret. Anything that restarts during the window will not start. Plan Vault maintenance as a change window for the portal’s deployments even though the portal itself stays up, and avoid scheduling node upgrades or autoscaling changes across the same period.

Is the injected file safe on the node?

It lives in an in-memory volume that is removed when the pod terminates, with permissions restricted to the container’s user — which is materially better than an environment variable, since it does not appear in process listings, crash dumps, or the diagnostic bundles that support tooling collects. It is still readable by anyone who can exec into the pod, so pod-exec permission remains a credential-equivalent grant.

Up one level: Secrets Management for Geospatial Platforms.