Geospatial Portal

Alerting on OGC Endpoint SLA Breaches

This guide sets up external, black-box alerting on a portal’s public OGC endpoints, so a broken GetCapabilities or a GetMap that returns an error body pages on-call before an agency user files a ticket.

It is a hands-on companion to Monitoring and Observability for Geospatial Portals and sits within the broader Infrastructure Orchestration & Configuration Management practice; the parent guide explains why internal metrics can read green while the outside world sees an outage, and why external probing is the only signal that matches the user’s experience. Here we build that probe and the burn-rate alerts on top of it.

Prerequisites

  • Prometheus 2.45+ with Alertmanager already receiving alerts and a working notification route (email, Slack, PagerDuty).
  • The blackbox_exporter (0.24.0+) deployed reachable from Prometheus, and able to reach the portal’s public URL — probe from outside the Kubernetes cluster’s internal network so the probe crosses the same ingress and TLS a real client does.
  • Defined SLOs for the endpoints, e.g. 99.9% availability and 95% of probes under 2s for the public WMS GetCapabilities.
  • Access to the portal’s public OGC endpoint URLs and any required TLS settings.
  • Permission to add scrape targets and rule files to Prometheus.

The flow below shows the prober issuing a real OGC request, Prometheus recording the probe result and burn rate, and a fast/slow burn-rate pair deciding whether Alertmanager pages.

Blackbox OGC probe to burn-rate recording rules to multi-window alert routing The blackbox exporter issues a real GetCapabilities or GetMap request across the public ingress to the OGC endpoint, then validates the HTTP status and matches the response body against an expected pattern. Prometheus scrapes the probe module, which yields a probe success value and a request duration. Recording rules turn those into an error-budget burn rate over both a short and a long window. A multi-window alerting rule fires only when both windows agree, and Alertmanager routes the page to on-call. OGC endpoint public WMS/WFS GetCapabilities via ingress + TLS blackbox_exporter probe module status 200 body regex match probe_success · duration Prometheus scrapes /probe recording: burn rate short + long window multi-window rule Alertmanager page on-call error budget 30-day window probe scrape fire

Step-by-step implementation

1. Define a blackbox probe module that checks a real OGC response

A bare HTTP 200 is not enough — a portal can return 200 with an OGC ServiceException body, or a login page, and still look “up.” The probe module must assert on the response body. This module issues a GET, requires a 2xx status, verifies the TLS certificate, and matches the XML that only a healthy GetCapabilities returns:

modules:
  ogc_getcapabilities:
    prober: http
    timeout: 10s
    http:
      method: GET
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      fail_if_not_ssl: true
      # A healthy capabilities document contains this element; an
      # exception report or an error page will not, so the probe fails closed.
      fail_if_body_not_matches_regexp:
        - "<(wms:)?WMS_Capabilities"
      fail_if_body_matches_regexp:
        - "ServiceException"

2. Add Prometheus probe scrape targets

Point Prometheus at the blackbox exporter with the target URLs as parameters. The relabel_configs idiom rewrites the scrape so __address__ becomes the exporter while the real endpoint travels as the target param and is preserved as the instance label.

scrape_configs:
  - job_name: blackbox-ogc
    metrics_path: /probe
    scrape_interval: 30s
    params:
      module: [ogc_getcapabilities]
    static_configs:
      - targets:
          - https://portal.example.gov/geoserver/wms?service=WMS&version=1.3.0&request=GetCapabilities
        labels:
          slo: wms-capabilities
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter.monitoring.svc:9115

3. Record availability and latency, then compute burn rate

Recording rules pre-aggregate the raw probe_success and probe_duration_seconds into the ratios the alerts read, so the alert expressions stay simple and every dashboard agrees on one series. Availability is the mean of probe_success; the burn rate is the failure fraction divided by the error budget, evaluated over two windows.

groups:
  - name: ogc-slo-recording
    interval: 30s
    rules:
      - record: 'slo:probe_availability:ratio_rate5m'
        expr: 'avg by (slo) (avg_over_time(probe_success[5m]))'
      - record: 'slo:probe_availability:ratio_rate1h'
        expr: 'avg by (slo) (avg_over_time(probe_success[1h]))'
      # Failure fraction / error budget (0.001 for a 99.9% target) = burn rate.
      - record: 'slo:error_budget_burn:rate5m'
        expr: '(1 - slo:probe_availability:ratio_rate5m) / 0.001'
      - record: 'slo:error_budget_burn:rate1h'
        expr: '(1 - slo:probe_availability:ratio_rate1h) / 0.001'
      - record: 'slo:probe_latency:p95_5m'
        expr: 'histogram_quantile(0.95, sum by (slo, le) (rate(probe_http_duration_seconds_bucket[5m])))'

4. Fire a multi-window, multi-burn-rate alert

A single-window alert either pages on every transient blip or reacts too slowly. The Google SRE multi-window pattern requires a fast window and a slow window to both exceed a threshold: a high burn rate confirmed over 5m and 1h means the budget is genuinely being consumed, not a one-off failed probe.

groups:
  - name: ogc-slo-alerts
    rules:
      - alert: OGCEndpointFastBurn
        expr: 'slo:error_budget_burn:rate5m > 14.4 and slo:error_budget_burn:rate1h > 14.4'
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Fast error-budget burn on {{ $labels.slo }}"
          description: "{{ $labels.slo }} is burning budget at {{ $value | printf \"%.1f\" }}x; the 30-day SLO is at risk within hours."
      - alert: OGCEndpointLatencySLO
        expr: 'slo:probe_latency:p95_5m > 2'
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "p95 probe latency SLO breach on {{ $labels.slo }}"
          description: "95th-percentile probe latency is {{ $value | printf \"%.2f\" }}s, over the 2s objective."

5. Route the alert through Alertmanager

Route by the severity label: page goes to the paging integration, ticket opens a lower-urgency notification. Group by slo so a single endpoint failing does not fan out into many pages.

route:
  receiver: default
  group_by: ["slo"]
  routes:
    - matchers: ['severity="page"']
      receiver: oncall-pager
      group_wait: 30s
      repeat_interval: 1h
    - matchers: ['severity="ticket"']
      receiver: ops-notifications
receivers:
  - name: default
    webhook_configs:
      - url: http://alert-sink.monitoring.svc:9099/
  - name: oncall-pager
    webhook_configs:
      - url: http://pager-bridge.monitoring.svc:8080/notify
  - name: ops-notifications
    webhook_configs:
      - url: http://ops-bridge.monitoring.svc:8080/notify

The endpoints being probed are the same ones fronted by the portal’s ingress, so pair this alerting with the caching and routing controls in Reverse Proxy Configuration for WMS/WFS — a probe that starts failing often points straight at a proxy change.

Choosing the window before choosing the threshold

An SLA alert has two numbers and teams usually argue about the wrong one. The threshold — 99.5% of requests under a second — is a policy decision that someone outside engineering can have an opinion about. The evaluation window is a technical decision that determines whether the alert is useful, and it is the one that decides how often you are woken for nothing.

A short window is sensitive and noisy: at one minute, a single slow batch of tiles from one client can breach a 99.5% target, so the alert fires during events no user noticed. A long window is quiet and late: at an hour, a total outage takes many minutes to move the ratio far enough to fire, because the healthy requests already recorded dilute the failing ones. Neither is wrong; they answer different questions, which is why mature alert sets carry both.

Fast and slow rules against the same error-budget target Three rows comparing a one-minute window, a one-hour window, and the pair used together, each with detection time for a total outage and the false-page behaviour on brief blips. RULE CATCHES A TOTAL OUTAGE IN ON A 30-SECOND BLIP 1-minute window sensitive about a minute pages — and nobody noticed the blip 1-hour window stable many minutes — diluted by healthy history silent, correctly Both, with different urgency fast rule pages, slow rule opens a ticket fast: about a minute slow: catches the drizzle a fast rule misses fast rule requires the breach to persist across two consecutive evaluations

The pairing matters because the two failure shapes are genuinely different. A total outage is caught by the fast rule. A slow degradation — a 2% failure rate that persists all day, well inside any single short window’s tolerance — is invisible to the fast rule and obvious to the slow one, and it is the shape that quietly consumes a month’s error budget without a single page. Give the fast rule a short “for” duration so a momentary spike must persist before it pages, and give the slow rule a lower urgency, because by construction it describes something that has already been happening for an hour and will still be there in ten minutes.

Verification 

# 1. The probe passes against a healthy endpoint (probe_success 1)
curl -s 'http://blackbox-exporter.monitoring.svc:9115/probe?module=ogc_getcapabilities&target=https://portal.example.gov/geoserver/wms?service=WMS%26request=GetCapabilities' \
  | grep '^probe_success'
#   expect: probe_success 1

# 2. Prometheus is scraping the blackbox job
curl -s 'http://prometheus.monitoring:9090/api/v1/query?query=probe_success{job="blackbox-ogc"}'

# 3. The burn-rate recording rule is populated
curl -s 'http://prometheus.monitoring:9090/api/v1/query?query=slo:error_budget_burn:rate5m'

# 4. Simulate a breach: point the target at a deliberately broken path and
#    confirm the alert enters Pending then Firing on the Prometheus /alerts page.

A probe_success 1 in step 1 with a matched body confirms the module asserts on content, not just status. If step 3 returns nothing, the recording rule file was not loaded — check rule_files in the Prometheus config and the /rules page.

Which endpoint the probe should call

An availability alert is only as honest as the request it makes, and for OGC services the easy choice is the misleading one. GetCapabilities is the obvious probe target — it is cheap, it needs no parameters, and it returns a document that is easy to assert on — and it is frequently served entirely from cache. A probe that only calls it will report a healthy portal while every renderer is down, because the one endpoint being checked never touches a renderer.

How deep each probe reaches into the stack Four probe types drawn as nested depths: capabilities reaching only the edge, warm tile reaching the cache, cold tile reaching the renderer, and filtered feature query reaching the database. Probe reach — each bar ends at the deepest component it proves edge cache renderer database GetCapabilities — often pure cache tile from a warm area — proves the cache path tile from a cold area — proves rendering filtered feature query — proves the whole path to the data

Run one probe of each depth and alert on them separately, because the pattern of which probes fail is the diagnosis. Capabilities up and cold tiles down points at the renderer; everything up except the feature query points at the database or its pool; all four down points at the edge or at DNS. A single composite “is the portal up” check collapses that information into one bit and hands the on-call engineer nothing but a starting position.

Two cautions on the deeper probes. Choose the cold-tile coordinates from an area with real data but negligible traffic, and rotate them, or the probe warms its own target and quietly becomes a cache check within a day. And keep the probe’s request cost small enough that its own load is irrelevant — a filtered query over a bounded extent, not a full-table scan that the database will feel every minute of every day.

Troubleshooting matrix

Symptom Likely cause Fix
probe_success is 0 for a healthy endpoint Body regex does not match the real capabilities document Fetch the document manually and adjust fail_if_body_not_matches_regexp to a stable element
Probe passes internally but users still see errors Probing from inside the Kubernetes cluster, bypassing ingress/TLS Run the exporter where it must cross the same public ingress a client does
Alert never fires despite outage Burn-rate threshold too high, or wrong error-budget divisor Verify the 0.001 divisor matches the SLO; lower-window threshold if the budget is smaller
Alert flaps on single failed probes Single-window rule or for: too short Use the two-window and expression and a for: 2m grace period
histogram_quantile latency rule empty probe_http_duration_seconds is a summary phase gauge, not a histogram Chart probe_duration_seconds directly, or enable the histogram export in the exporter
Page fires but no notification Alertmanager route has no matching receiver Check the route tree on the Alertmanager UI; confirm the severity matcher matches the alert label

Up one level: Monitoring and Observability for Geospatial Portals.