Geospatial Portal

Autoscaling Tile Renderers with Custom Metrics

This guide configures a renderer fleet to scale on signals that move when readers are affected — in-flight requests and cache-miss ratio — rather than on CPU. It belongs to Autoscaling & Capacity Planning for Geospatial Workloads, within the Infrastructure Orchestration & Configuration Management framework.

Prerequisites

  • Kubernetes 1.27 or newer with a metrics adapter that exposes custom metrics to the autoscaler.
  • Prometheus scraping the renderer and the tile proxy, per exporting GeoServer metrics to Prometheus.
  • A measured concurrency-versus-latency curve for one renderer, which the target below comes from.
  • A known connection ceiling at the pooler, which the fleet’s maximum is derived from.

Find the target before configuring the scaler

The target is not a preference; it is a measurement. Run one renderer at increasing concurrency against representative traffic and record p95 latency, and the useful target is the concurrency just below the point where latency starts to climb.

Where the autoscaling target sits on the latency curve A latency-versus-concurrency curve with a flat region, a knee, and a steep region, with the target marked just below the knee. concurrent requests per renderer → p95 latency target just below the knee latency starts to climb here beyond it, throughput is flat and every reader waits longer idle capacity

Step-by-step implementation

1. Expose the two signals as scalable metrics 

# prometheus-adapter values: expose per-pod in-flight requests and a fleet-wide
# cache-miss ratio to the Kubernetes custom metrics API.
rules:
  custom:
    # In-flight requests per renderer pod — the primary scaling signal.
    - seriesQuery: 'geoserver_requests_in_flight{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod:       {resource: "pod"}
      name: {as: "renderer_inflight_requests"}
      metricsQuery: 'avg_over_time(<<.Series>>{<<.LabelMatchers>>}[2m])'

  external:
    # Cache-miss ratio at the proxy — the leading signal, fleet-wide.
    - seriesQuery: 'nginx_cache_status_total{status="MISS"}'
      name: {as: "tile_cache_miss_ratio"}
      metricsQuery: |
        sum(rate(nginx_cache_status_total{status="MISS"}[5m]))
        /
        sum(rate(nginx_cache_status_total[5m]))

2. Configure the scaler with both signals and asymmetric windows 

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: tile-renderer
  namespace: geoportal
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tile-renderer

  # The floor absorbs the first seconds of a surge; the cap is derived from the
  # database pool, NOT from a budget round number.
  minReplicas: 4
  maxReplicas: 20

  metrics:
    - type: Pods
      pods:
        metric: {name: renderer_inflight_requests}
        target:
          type: AverageValue
          averageValue: "12"          # measured, just below the knee

    - type: External
      external:
        metric: {name: tile_cache_miss_ratio}
        target:
          type: Value
          value: "0.25"               # sustained misses mean renderer load is coming

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100                  # double quickly; tile surges are steep
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600 # slow, so a lull does not strand the next burst
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120

3. Cap the fleet from the downstream ceiling 

# The maximum replica count is a function of what the database can serve, not of
# what the cluster can schedule. Derive it rather than choosing it.
POOL_MAX=$(psql -Atc "SHOW max_connections")           # e.g. 200
RESERVED=20                                            # maintenance + monitoring
PER_RENDERER=8                                         # pool size per renderer pod
echo $(( (POOL_MAX - RESERVED) / PER_RENDERER ))
#   this number is maxReplicas — beyond it, renderers wait for connections

4. Drain cleanly on scale-down 

spec:
  template:
    spec:
      # Longer than the p99 render, so an in-flight tile is never killed.
      terminationGracePeriodSeconds: 90
      containers:
        - name: renderer
          lifecycle:
            preStop:
              exec:
                # Fail readiness first so the proxy stops routing, then wait for
                # in-flight work to finish before the process is signalled.
                command: ["/bin/sh", "-c", "touch /tmp/draining && sleep 20"]
          readinessProbe:
            exec:
              command: ["/bin/sh", "-c", "! test -f /tmp/draining && curl -fsS localhost:8080/healthz"]
            periodSeconds: 5
Scale-down timing, and the number that must be largest A timeline from the drain marker through readiness failure and in-flight completion to process termination, with the grace period spanning all of it. One pod being removed mark draining readiness fails proxy stops routing in-flight renders complete terminationGracePeriodSeconds must span all of the above The failure this prevents A grace period shorter than the drain kills the process while the proxy is still waiting for tiles it requested, so a routine scale-down produces a burst of failed tiles — and tile clients retry, arriving just as capacity fell. Set the grace period from the measured p99 render time, and re-check it whenever cartography changes.

Watch the scaler itself, not only what it scales

An autoscaler is a control loop, and like any control loop it can be wrong in ways that are invisible from the thing it controls. A fleet sitting at a comfortable size might be there because demand is comfortable, or because the metric pipeline stopped reporting an hour ago and the scaler is holding the last value it saw.

Three things to watch about the scaler Metric age, scaling event rate, and time at maximum, each with what it reveals and why the portal's own metrics do not show it. metric age how old is the value the scaler is acting on? reveals: a stalled pipeline the fleet freezes at its last size and nothing errors events per hour how often does it change the replica count? reveals: oscillation a fleet that never settles is always partly cold time at maximum how long is it pinned at the cap? reveals: the cap is the constraint raise the downstream ceiling, not the cap on its own

Alert on the first of these most loudly. A stalled metric pipeline is indistinguishable from a stable workload from every other angle, and it produces its consequence — a fleet that does not grow — at exactly the moment growth is needed.

Verification 

# 1. The custom metrics are visible to the autoscaler
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/geoportal/pods/*/renderer_inflight_requests" | jq .
#   expect: a value per renderer pod, not an error

# 2. The scaler reports both metrics, and its current target
kubectl -n geoportal describe hpa tile-renderer | sed -n '/Metrics/,/Events/p'
#   expect: both metrics listed with current and target values

# 3. Load causes a scale-up within the stabilisation window
k6 run --vus 200 --duration 3m tile-load.js &
watch -n5 'kubectl -n geoportal get hpa tile-renderer'
#   expect: replicas climbing within about a minute of the metric crossing target

# 4. Scale-down does not drop requests
kubectl -n geoportal scale deploy/tile-renderer --replicas=4
#   during the drain, error count in the proxy log should remain zero

# 5. The cap holds, and pool waits stay near zero at the cap
kubectl -n geoportal get hpa tile-renderer -o jsonpath='{.spec.maxReplicas}{"\n"}'
psql -h pgbouncer -p 6432 -U pgbouncer -c "SHOW POOLS;" | awk '{print $1, $6, $7}'
#   expect: cl_waiting near zero when the fleet is at maxReplicas

Check 5 is the one that validates the cap. If clients are waiting for connections while the fleet is at its maximum, the cap is too high and the renderer fleet is converting a database limit into a latency problem.

Troubleshooting matrix

Symptom Likely cause Fix
The fleet never scales despite obvious load The custom metric is not reaching the metrics API Query the raw metrics endpoint; check adapter rules and label matching
Replicas oscillate between two values Target too close to steady state, windows too short Widen the scale-down stabilisation window; set the target below the knee
Scaling happens after readers complain Scaling on CPU, or a long metric averaging window Use in-flight requests with a two-minute average; add the miss ratio
Failed tiles appear during every scale-down Grace period shorter than the drain Set grace period above p99 render; fail readiness before terminating
The fleet reaches its cap and latency stays high The bottleneck is the connection pool, not the renderer Confirm with pool wait time; raise the pool or the database, not the cap
Scale-up doubles the fleet unnecessarily on a brief spike Percent policy of 100 with a short stabilisation window Keep the aggressive policy but lengthen the window slightly
Cost rose after enabling autoscaling The floor was set from peak rather than from surge absorption Size the floor from how fast traffic climbs, not from average demand

FAQ

Why include cache-miss ratio when in-flight requests already reacts?

Because it moves first. Renderer load is a consequence of misses, so a rising miss ratio — after an invalidation, or when a new area becomes popular — predicts the load before any renderer feels it. Using it alongside the reactive signal buys roughly the fleet’s start-up time, which is exactly the interval that decides whether readers notice.

Should the two metrics be combined or evaluated separately?

Separately. The autoscaler takes the maximum of the replica counts each metric implies, which is the behaviour you want: either signal alone is sufficient reason to grow, and neither should be able to hold the fleet down while the other is alarming.

What if the renderer does not expose in-flight requests?

Derive it at the proxy instead, as active upstream connections per backend, and expose that per pod. It is a slightly less precise signal because it counts connections rather than renders, and it is close enough — and it works for any renderer, including ones you did not build.

Is a floor of four replicas not wasteful for a small portal?

It is a deliberate purchase of surge absorption. The right floor is the capacity needed to serve the first thirty seconds of the fastest realistic traffic climb, which for most agency portals is a small number of replicas and a small absolute cost. Sizing the floor from average demand is what produces the slow morning that gets autoscaling disabled.

How does this interact with node autoscaling?

It adds the node provisioning time to every scale-up that does not fit on existing nodes, which can be minutes rather than seconds. Keep enough scheduled headroom — or a small over-provisioned buffer of low-priority placeholder pods — that a renderer scale-up rarely waits for a node.

What should happen when the metrics adapter itself is unavailable?

The autoscaler holds the fleet at its current size, which is a safe default and a silent one. Treat adapter availability as a first-class dependency: alert on the age of the metric the scaler is reading, and set the floor high enough that a frozen fleet can still serve a normal working day. A portal whose floor is one replica and whose adapter fails overnight arrives at the morning peak with one renderer and no mechanism to notice.

Does this apply to vector tile servers as well?

Yes, with one adjustment: vector tile generation cost tracks feature density rather than pixel area, so the concurrency-versus-latency curve differs sharply between a sparse rural extent and a dense city. Measure the curve against the densest part of the coverage, because that is where the fleet will be under pressure and where the knee is lowest.

Up one level: Autoscaling & Capacity Planning for Geospatial Workloads.