Geospatial Portal

Monitoring and Observability for Geospatial Portals

A geospatial portal rarely dies loudly. It degrades: a MapProxy cache goes cold after a redeploy, GetMap latency creeps past a browser timeout, a Celery ingestion queue backs up until harvest jobs silently expire, and the first anyone hears of it is an agency user reporting “the map is blank.” Without telemetry on each moving part, operators are left reading tea leaves in a proxy access log after the outage is already public.

This guide sits within the Infrastructure Orchestration & Configuration Management practice and lays out how to instrument an open-source GIS stack — GeoServer, MapProxy, PostGIS, Celery, and the public OGC endpoints — with the three telemetry signals, then how to run a Prometheus, Grafana, Loki, and Alertmanager pipeline that turns those signals into dashboards, service-level objectives, and pages that fire before users notice.

The pipeline below shows where each collector runs, what it scrapes or tails from the GIS components, and how the signals converge on Grafana and Alertmanager.

Metrics, logs, and probe signals converging on Prometheus, Loki, Grafana, and Alertmanager On the left, five geospatial components emit telemetry: GeoServer exposes JVM and request metrics through a JMX exporter, MapProxy reports cache hit ratios, PostGIS exports connection and slow-query stats, Celery emits queue depth metrics and structured JSON logs, and a blackbox exporter probes public OGC endpoints. In the middle, Prometheus scrapes all metrics endpoints on an interval while Promtail tails Celery and component logs into Loki. On the right, Grafana reads from both Prometheus and Loki to render dashboards, and Prometheus evaluates alerting rules that fan out through Alertmanager to on-call routes. GIS COMPONENTS GeoServerJVM heap · req latency MapProxycache hit ratio PostGISconnections · slow queries Celery workersqueue depth · JSON logs Blackbox proberGetCapabilities · GetMap Prometheus scrape + rule engine RED / USE series recording + alert rules Promtail / Alloy tails + labels logs → Loki scrape tail logs Loki LogQL store Grafana dashboards · SLOs Alert- manager on-call routes fires alerts

The Three Signals Mapped to a GIS Stack

Observability rests on three complementary signals, and a portal needs all three because each answers a different question during an incident.

Metrics are cheap, numeric time series — request counts, latencies, heap bytes, queue lengths — sampled on a fixed interval. They answer is the system healthy right now, and how does that compare to an hour ago? They are what dashboards and alerts run on. Logs are discrete, high-cardinality events with context — a stack trace, a failing task id, the exact ServiceException a WFS client received. They answer what exactly happened to this one request? Traces stitch a single request across component boundaries — an OpenLayers GetMap that fans into a GeoServer render, a PostGIS ST_AsMVT call, and a MapProxy cache lookup — and answer which hop in the chain spent the time?

The practical rule for a small operations team is to instrument metrics everywhere first, ship structured logs second, and add tracing selectively to the request paths where cross-component latency is genuinely ambiguous. Metrics tell you that the portal is slow; logs and traces tell you why. A GIS stack punishes teams that skip the first step, because the most damaging failures — a cold cache, a saturated connection pool — are invisible in any single component’s log but obvious in a metric trend.

What to Instrument Per Component

Instrumentation is not uniform: each component fails in its own idiom, so each needs its own golden signals.

  • GeoServer runs on the JVM, so the two failure modes are heap pressure and slow rendering. Instrument JVM heap used vs. committed, garbage-collection pause time, live thread count, and per-request throughput and latency broken out by OGC operation (GetMap, GetFeature, GetCapabilities). A JMX exporter surfaces all of this; the mechanics live in Exporting GeoServer Metrics to Prometheus.
  • MapProxy lives or dies on its cache hit ratio. A ratio that drops after a deploy means the tile cache was wiped and every request is now a cache-miss render against GeoServer — a self-inflicted load spike. Track hits, misses, and the derived ratio, plus seeding progress for large extents.
  • PostGIS is the shared floor everything else stands on. Instrument active vs. idle connections against max_connections, lock waits, replication lag, and slow spatial-query counts. A pool nearing its ceiling is an outage in waiting; sizing it correctly is the subject of Optimizing PostgreSQL/PostGIS Connection Limits.
  • Celery ingestion and harvesting pipelines fail by backing up. Instrument per-queue depth, task latency percentiles, and success/failure/retry counts. Rising queue depth with flat throughput means workers are starved or wedged. Structured logs make the failing tasks searchable — see Aggregating Celery Logs with Loki.
  • Public OGC endpoints need external truth. Internal metrics can all look green while a broken TLS chain or a bad ingress rule makes GetCapabilities unreachable from the outside. A blackbox prober that issues a real OGC request from outside the Kubernetes cluster is the only signal that reflects the user’s actual experience; wiring it to SLO alerts is covered in Alerting on OGC Endpoint SLA Breaches.

Where Each Piece of the Stack Runs

The collection stack is itself a small distributed system, and placing its parts wrong is a common way to lose visibility exactly when you need it. Prometheus runs as a central scraper — one replica per cluster is usually enough for a portal, sized by the number of series it holds in its head block. It pulls metrics from each component’s exporter on a fixed scrape_interval (15s is a sane default for GIS traffic), evaluates recording and alerting rules, and holds recent data locally while shipping long-term samples to remote storage.

Log collection is push-adjacent: a Promtail or Grafana Alloy agent runs as a DaemonSet — one instance per node — tailing container stdout, attaching labels (namespace, pod, queue), and forwarding lines to Loki. Loki itself runs centrally alongside Prometheus, indexing only the labels rather than the full log text, which is what keeps it cheap enough to retain weeks of Celery logs.

Grafana runs as a stateless read tier with Prometheus and Loki configured as data sources, so a single dashboard panel can overlay a latency metric with the log lines from the same window. Alertmanager runs as a separate deduplicating, grouping, and routing layer that Prometheus pushes fired alerts into — never inline in Prometheus itself, so that alert delivery survives a Prometheus restart. Crucially, the entire stack runs outside the blast radius of the portal it watches: put it in its own namespace with its own PostGIS-free storage, or a portal outage takes down the tooling meant to diagnose it. That isolation mirrors the boundary discipline the data tier already enforces in Kubernetes StatefulSets for PostGIS Databases.

Applying RED and USE to Tile and OGC Traffic

Two complementary methods keep instrumentation disciplined instead of a sprawl of ad-hoc gauges. The RED method — Rate, Errors, Duration — models request-driven services from the caller’s perspective and is the right lens for OGC and tile traffic. The USE method — Utilization, Saturation, Errors — models finite resources and is the right lens for the pools and queues underneath.

For a tile endpoint, RED translates directly: Rate is GetMap/tile requests per second, Errors is the fraction returning a 5xx or an OGC ServiceException, and Duration is the request-latency histogram from which you read p50 and p99. A single Grafana row per endpoint carrying those three panels tells an operator, at a glance, whether the tile path is healthy. The example below defines a recording rule that pre-computes the tile error ratio so dashboards and alerts share one authoritative series:

groups:
  - name: gis-red-tiles
    rules:
      - record: portal:tile_requests:rate5m
        expr: 'sum by (endpoint) (rate(geoserver_requests_total{operation="GetMap"}[5m]))'
      - record: portal:tile_errors:ratio5m
        expr: 'sum by (endpoint) (rate(geoserver_requests_total{operation="GetMap",status=~"5.."}[5m]))
               / sum by (endpoint) (rate(geoserver_requests_total{operation="GetMap"}[5m]))'
      - record: portal:tile_latency:p99_5m
        expr: 'histogram_quantile(0.99, sum by (endpoint, le) (rate(geoserver_request_duration_seconds_bucket{operation="GetMap"}[5m])))'

USE covers what RED cannot see. The PostGIS connection pool has a Utilization (active connections / max_connections), a Saturation (clients waiting for a free connection), and Errors (too many connections refusals). The Celery queue has Utilization (busy workers / total), Saturation (queue depth), and Errors (task failures). When a tile endpoint’s RED Duration spikes, the USE metrics of the pool and queue beneath it are where the cause almost always shows.

Dashboards and SLO Design

Instrumentation without a target is just decoration. A service-level objective turns “is the portal fast?” into a measurable contract: for example, 99% of GetMap requests complete under 800ms over a rolling 30-day window, and the public WMS GetCapabilities endpoint is available 99.9% of the time. The SLO defines an error budget — the 1% you are allowed to burn — and the alerting philosophy follows from it: page a human only when the budget is burning fast enough to be exhausted before the window closes, not on every transient blip.

Dashboards should be layered to match how an incident is actually worked. A top-level Grafana overview shows one SLO-status tile and one RED row per public endpoint — the view an on-call engineer opens first. Component dashboards sit one layer down: a GeoServer JVM board, a MapProxy cache board, a PostGIS board, a Celery board, each carrying that component’s USE metrics. The drill path is deliberate: SLO breach on the overview → the endpoint’s RED row → the underlying component’s USE board → the correlated logs in Loki. Building the boards in that order, rather than dumping every metric onto one wall of graphs, is what makes the pipeline usable at 3 a.m.

Provision the dashboards and rules as code, not by clicking in the UI. Store Grafana dashboard JSON and Prometheus rule YAML in the same GitOps repository that reconciles the portal, so a monitoring change is reviewed, versioned, and rolled back like any other — the parity discipline described in Environment Parity in Geospatial CI Pipelines applies to the observability stack as much as to the portal.

Operational Troubleshooting

The pipeline itself fails in recognizable ways; work the matrix symptom-first before doubting the portal.

  • A target shows down in Prometheus but the component is clearly serving traffic. The scrape endpoint or port is wrong, or a NetworkPolicy blocks Prometheus from reaching the exporter. Check up{job="..."} and the target’s Last Error on the Prometheus targets page; confirm the exporter port is reachable from the Prometheus namespace.
  • Grafana panels read No data while Prometheus has the series. The panel’s PromQL uses a label the recording rule doesn’t emit, or the dashboard points at the wrong data source. Run the query in Prometheus directly to confirm the series name and labels.
  • Alerts fire but no notification arrives. Prometheus is evaluating the rule but Alertmanager routing or the receiver is misconfigured. Check the Alertmanager status page for the active alert and inspect the route tree; a missing receiver match silently drops the page.
  • MapProxy cache-hit ratio reads 100% and never moves. The exporter is reporting a cached counter or the metric is only updated on eviction. Confirm the counter increments under live traffic before trusting the panel.
  • Loki queries are slow or time out. A LogQL query filters on log content rather than labels, forcing a full scan. Add a label matcher ({queue="ingest"}) to narrow the stream before the line-filter, and confirm Promtail is labelling by queue and worker.
  • Prometheus memory climbs until it OOMs. Metric cardinality has exploded — usually a label carrying an unbounded value like a full request URL or a task id. Find the offender with topk(10, count by (__name__)({__name__=~".+"})) and drop the high-cardinality label at the exporter.

Treated as a first-class part of the platform rather than an afterthought bolted on after the first outage, this pipeline converts a geospatial portal from a system that fails silently into one that announces its own degradation — with enough context for a small team to meet public-sector availability commitments without staring at logs all night.

Up one level: Infrastructure Orchestration & Configuration Management.