Geospatial Portal

Monitoring and Observability for Geospatial Portals

A geospatial portal rarely dies loudly. It degrades: a MapProxy cache goes cold after a redeploy, GetMap latency creeps past a browser timeout, a Celery ingestion queue backs up until harvest jobs silently expire, and the first anyone hears of it is an agency user reporting “the map is blank.” Without telemetry on each moving part, operators are left reading tea leaves in a proxy access log after the outage is already public.

This guide sits within the Infrastructure Orchestration & Configuration Management practice and lays out how to instrument an open-source GIS stack — GeoServer, MapProxy, PostGIS, Celery, and the public OGC endpoints — with the three telemetry signals, then how to run a Prometheus, Grafana, Loki, and Alertmanager pipeline that turns those signals into dashboards, service-level objectives, and pages that fire before users notice.

The pipeline below shows where each collector runs, what it scrapes or tails from the GIS components, and how the signals converge on Grafana and Alertmanager.

Metrics, logs, and probe signals converging on Prometheus, Loki, Grafana, and Alertmanager On the left, five geospatial components emit telemetry: GeoServer exposes JVM and request metrics through a JMX exporter, MapProxy reports cache hit ratios, PostGIS exports connection and slow-query stats, Celery emits queue depth metrics and structured JSON logs, and a blackbox exporter probes public OGC endpoints. In the middle, Prometheus scrapes all metrics endpoints on an interval while Promtail tails Celery and component logs into Loki. On the right, Grafana reads from both Prometheus and Loki to render dashboards, and Prometheus evaluates alerting rules that fan out through Alertmanager to on-call routes. GIS COMPONENTS GeoServerJVM heap · req latency MapProxycache hit ratio PostGISconnections · slow queries Celery workersqueue depth · JSON logs Blackbox proberGetCapabilities · GetMap Prometheus scrape + rule engine RED / USE series recording + alert rules Promtail / Alloy tails + labels logs → Loki scrape tail logs Loki LogQL store Grafana dashboards · SLOs Alert- manager on-call routes fires alerts

The Three Signals Mapped to a GIS Stack

Observability rests on three complementary signals, and a portal needs all three because each answers a different question during an incident.

Metrics are cheap, numeric time series — request counts, latencies, heap bytes, queue lengths — sampled on a fixed interval. They answer is the system healthy right now, and how does that compare to an hour ago? They are what dashboards and alerts run on. Logs are discrete, high-cardinality events with context — a stack trace, a failing task id, the exact ServiceException a WFS client received. They answer what exactly happened to this one request? Traces stitch a single request across component boundaries — an OpenLayers GetMap that fans into a GeoServer render, a PostGIS ST_AsMVT call, and a MapProxy cache lookup — and answer which hop in the chain spent the time?

The practical rule for a small operations team is to instrument metrics everywhere first, ship structured logs second, and add tracing selectively to the request paths where cross-component latency is genuinely ambiguous. Metrics tell you that the portal is slow; logs and traces tell you why. A GIS stack punishes teams that skip the first step, because the most damaging failures — a cold cache, a saturated connection pool — are invisible in any single component’s log but obvious in a metric trend.

What to Instrument Per Component

Instrumentation is not uniform: each component fails in its own idiom, so each needs its own golden signals.

  • GeoServer runs on the JVM, so the two failure modes are heap pressure and slow rendering. Instrument JVM heap used vs. committed, garbage-collection pause time, live thread count, and per-request throughput and latency broken out by OGC operation (GetMap, GetFeature, GetCapabilities). A JMX exporter surfaces all of this; the mechanics live in Exporting GeoServer Metrics to Prometheus.
  • MapProxy lives or dies on its cache hit ratio. A ratio that drops after a deploy means the tile cache was wiped and every request is now a cache-miss render against GeoServer — a self-inflicted load spike. Track hits, misses, and the derived ratio, plus seeding progress for large extents.
  • PostGIS is the shared floor everything else stands on. Instrument active vs. idle connections against max_connections, lock waits, replication lag, and slow spatial-query counts. A pool nearing its ceiling is an outage in waiting; sizing it correctly is the subject of Optimizing PostgreSQL/PostGIS Connection Limits.
  • Celery ingestion and harvesting pipelines fail by backing up. Instrument per-queue depth, task latency percentiles, and success/failure/retry counts. Rising queue depth with flat throughput means workers are starved or wedged. Structured logs make the failing tasks searchable — see Aggregating Celery Logs with Loki.
  • Public OGC endpoints need external truth. Internal metrics can all look green while a broken TLS chain or a bad ingress rule makes GetCapabilities unreachable from the outside. A blackbox prober that issues a real OGC request from outside the Kubernetes cluster is the only signal that reflects the user’s actual experience; wiring it to SLO alerts is covered in Alerting on OGC Endpoint SLA Breaches.

Where Each Piece of the Stack Runs

The collection stack is itself a small distributed system, and placing its parts wrong is a common way to lose visibility exactly when you need it. Prometheus runs as a central scraper — one replica per cluster is usually enough for a portal, sized by the number of series it holds in its head block. It pulls metrics from each component’s exporter on a fixed scrape_interval (15s is a sane default for GIS traffic), evaluates recording and alerting rules, and holds recent data locally while shipping long-term samples to remote storage.

Log collection is push-adjacent: a Promtail or Grafana Alloy agent runs as a DaemonSet — one instance per node — tailing container stdout, attaching labels (namespace, pod, queue), and forwarding lines to Loki. Loki itself runs centrally alongside Prometheus, indexing only the labels rather than the full log text, which is what keeps it cheap enough to retain weeks of Celery logs.

Grafana runs as a stateless read tier with Prometheus and Loki configured as data sources, so a single dashboard panel can overlay a latency metric with the log lines from the same window. Alertmanager runs as a separate deduplicating, grouping, and routing layer that Prometheus pushes fired alerts into — never inline in Prometheus itself, so that alert delivery survives a Prometheus restart. Crucially, the entire stack runs outside the blast radius of the portal it watches: put it in its own namespace with its own PostGIS-free storage, or a portal outage takes down the tooling meant to diagnose it. That isolation mirrors the boundary discipline the data tier already enforces in Kubernetes StatefulSets for PostGIS Databases.

Applying RED and USE to Tile and OGC Traffic

Two complementary methods keep instrumentation disciplined instead of a sprawl of ad-hoc gauges. The RED method — Rate, Errors, Duration — models request-driven services from the caller’s perspective and is the right lens for OGC and tile traffic. The USE method — Utilization, Saturation, Errors — models finite resources and is the right lens for the pools and queues underneath.

For a tile endpoint, RED translates directly: Rate is GetMap/tile requests per second, Errors is the fraction returning a 5xx or an OGC ServiceException, and Duration is the request-latency histogram from which you read p50 and p99. A single Grafana row per endpoint carrying those three panels tells an operator, at a glance, whether the tile path is healthy. The example below defines a recording rule that pre-computes the tile error ratio so dashboards and alerts share one authoritative series:

groups:
  - name: gis-red-tiles
    rules:
      - record: portal:tile_requests:rate5m
        expr: 'sum by (endpoint) (rate(geoserver_requests_total{operation="GetMap"}[5m]))'
      - record: portal:tile_errors:ratio5m
        expr: 'sum by (endpoint) (rate(geoserver_requests_total{operation="GetMap",status=~"5.."}[5m]))
               / sum by (endpoint) (rate(geoserver_requests_total{operation="GetMap"}[5m]))'
      - record: portal:tile_latency:p99_5m
        expr: 'histogram_quantile(0.99, sum by (endpoint, le) (rate(geoserver_request_duration_seconds_bucket{operation="GetMap"}[5m])))'

USE covers what RED cannot see. The PostGIS connection pool has a Utilization (active connections / max_connections), a Saturation (clients waiting for a free connection), and Errors (too many connections refusals). The Celery queue has Utilization (busy workers / total), Saturation (queue depth), and Errors (task failures). When a tile endpoint’s RED Duration spikes, the USE metrics of the pool and queue beneath it are where the cause almost always shows.

Cardinality: the constraint that shapes every label decision

Geospatial telemetry has an unusual failure mode, and teams meet it about three weeks after the first dashboard goes up. The obvious labels to attach to a tile metric — layer, zoom, x, y, tenant, format, CRS — are exactly the labels that destroy a time-series database, because every distinct combination is a separate series held in memory for as long as it is active. A single layer served across twelve zoom levels for forty tenants in two formats is already 960 series; adding tile coordinates makes it unbounded, and a metrics backend that was comfortable on Monday is out of memory on Friday with no change to the portal at all.

What may be a metric label, and what belongs in logs instead Four bands of label cardinality from safe to forbidden, each with example labels, the rough series count, and where the excluded detail should live instead. Safe service, method, status class, cache result tens of series bounded by the design of the system, not by traffic Useful layer, zoom band (not exact zoom), format hundreds of series group zooms into low, mid and deep rather than one label per level Careful tenant — multiplies every other label thousands, and grows keep per-tenant series for the top N; bucket the rest as “other” Never tile x/y, full request path, user id, bbox unbounded these belong in structured logs and traces, joined by request id

The bottom row is not a loss of information — it is a relocation. The question “which tile was slow” is a log or trace question, answered by finding the request and reading its fields; the question “are deep-zoom tiles slower than shallow ones” is a metric question, answered by a zoom-band label with three values. Deciding which system answers which question before instrumenting is what keeps both affordable, and retrofitting the split after the metrics backend has fallen over is considerably more disruptive than choosing it up front.

Dashboards and SLO Design

Instrumentation without a target is just decoration. A service-level objective turns “is the portal fast?” into a measurable contract: for example, 99% of GetMap requests complete under 800ms over a rolling 30-day window, and the public WMS GetCapabilities endpoint is available 99.9% of the time. The SLO defines an error budget — the 1% you are allowed to burn — and the alerting philosophy follows from it: page a human only when the budget is burning fast enough to be exhausted before the window closes, not on every transient blip.

Dashboards should be layered to match how an incident is actually worked. A top-level Grafana overview shows one SLO-status tile and one RED row per public endpoint — the view an on-call engineer opens first. Component dashboards sit one layer down: a GeoServer JVM board, a MapProxy cache board, a PostGIS board, a Celery board, each carrying that component’s USE metrics. The drill path is deliberate: SLO breach on the overview → the endpoint’s RED row → the underlying component’s USE board → the correlated logs in Loki. Building the boards in that order, rather than dumping every metric onto one wall of graphs, is what makes the pipeline usable at 3 a.m.

Provision the dashboards and rules as code, not by clicking in the UI. Store Grafana dashboard JSON and Prometheus rule YAML in the same GitOps repository that reconciles the portal, so a monitoring change is reviewed, versioned, and rolled back like any other — the parity discipline described in Environment Parity in Geospatial CI Pipelines applies to the observability stack as much as to the portal.

Alerting on the user’s experience, not the component’s mood

The most common alerting mistake in a portal is symmetrical to the cardinality mistake: alerting on everything that can be measured rather than on the small number of conditions that mean a person cannot do their work. A rule that fires when renderer CPU passes eighty percent will fire during every seeding run and every busy afternoon, and within a month it is muted. A rule that fires when the share of tile requests slower than one second exceeds a threshold for five minutes fires when the map is actually slow, which is rarely, and is therefore still trusted a year later.

Anchor the alert set to a handful of user-visible statements and derive everything else from them. For a geospatial portal, four cover the ground: the map draws, the catalogue answers, published data is current, and edits are accepted. Each of those maps onto a measurable ratio with a window, and each has an obvious owner during an incident.

Four promises to users, and the signal that proves each Four rows, each pairing a plain-language promise with the ratio that measures it, the evaluation window, and the user-visible meaning of a breach. PROMISE SIGNAL A BREACH MEANS The map draws share of tiles served under 1s, 5-minute window across all tiers, measured at the edge blank or half-drawn viewports The catalogue answers search success rate and p95 latency including the empty-result path nobody can find a dataset Data is current age of the last successful publish or harvest per source, compared to its cadence confident decisions on stale data Edits are accepted feature transaction success rate field work is silently lost

The third row is the one most portals lack and the one that catches the quietest failures. A harvest that stops running, a publish job that fails on one source, a seeding pipeline that has not completed since a certificate expired — none of these change a latency graph or produce an error rate, because nothing is failing in the request path. What changes is the age of the newest record, and a rule comparing that age against each source’s expected cadence turns an invisible, indefinite staleness into a page on the first missed cycle. Component-level alerts still have a place beneath these, but as diagnostic context during an incident rather than as the things that wake somebody up.

Retention: keeping enough history to answer seasonal questions

Geospatial portals have seasons. Flood layers are consulted in winter, agricultural parcels around subsidy deadlines, election boundaries in the weeks before a poll, and wildfire perimeters in a window that moves with the climate. A metrics retention window of two weeks — a common default — cannot answer the question that capacity planning actually asks, which is “what did this look like the last time this season came round”. The result is that every seasonal peak is met as a novelty, and the same emergency scaling conversation happens annually.

Resolve this with tiered retention rather than by keeping everything at full resolution. High-resolution samples are needed only for incident debugging, where the useful window is hours to days. Downsampled series — a few key ratios at five-minute or hourly resolution — are what capacity planning needs, and they are cheap enough to keep for years. Choose the small set of series that deserve long retention deliberately: request rate per service, the four user-facing ratios above, cache hit ratio, and the saturation signal for each tier. Everything else can expire quickly without loss, because its job was to explain an incident that has since been resolved.

Keep one more thing alongside the numbers: a short dated note for each significant event — a release, a seeding change, a new data source, an incident. A year later the series alone will show a step change with no explanation, and the note is what turns an anomaly into a known cause. It costs a line of text and it is the difference between history and a graph.

Operational Troubleshooting

The pipeline itself fails in recognizable ways; work the matrix symptom-first before doubting the portal.

  • A target shows down in Prometheus but the component is clearly serving traffic. The scrape endpoint or port is wrong, or a NetworkPolicy blocks Prometheus from reaching the exporter. Check up{job="..."} and the target’s Last Error on the Prometheus targets page; confirm the exporter port is reachable from the Prometheus namespace.
  • Grafana panels read No data while Prometheus has the series. The panel’s PromQL uses a label the recording rule doesn’t emit, or the dashboard points at the wrong data source. Run the query in Prometheus directly to confirm the series name and labels.
  • Alerts fire but no notification arrives. Prometheus is evaluating the rule but Alertmanager routing or the receiver is misconfigured. Check the Alertmanager status page for the active alert and inspect the route tree; a missing receiver match silently drops the page.
  • MapProxy cache-hit ratio reads 100% and never moves. The exporter is reporting a cached counter or the metric is only updated on eviction. Confirm the counter increments under live traffic before trusting the panel.
  • Loki queries are slow or time out. A LogQL query filters on log content rather than labels, forcing a full scan. Add a label matcher ({queue="ingest"}) to narrow the stream before the line-filter, and confirm Promtail is labelling by queue and worker.
  • Prometheus memory climbs until it OOMs. Metric cardinality has exploded — usually a label carrying an unbounded value like a full request URL or a task id. Find the offender with topk(10, count by (__name__)({__name__=~".+"})) and drop the high-cardinality label at the exporter.

Treated as a first-class part of the platform rather than an afterthought bolted on after the first outage, this pipeline converts a geospatial portal from a system that fails silently into one that announces its own degradation — with enough context for a small team to meet public-sector availability commitments without staring at logs all night.

Up one level: Infrastructure Orchestration & Configuration Management.