Alerting on OGC Endpoint SLA Breaches

This guide sets up external, black-box alerting on a portal’s public OGC endpoints, so a broken GetCapabilities or a GetMap that returns an error body pages on-call before an agency user files a ticket.

It is a hands-on companion to Monitoring and Observability for Geospatial Portals and sits within the broader Infrastructure Orchestration & Configuration Management practice; the parent guide explains why internal metrics can read green while the outside world sees an outage, and why external probing is the only signal that matches the user’s experience. Here we build that probe and the burn-rate alerts on top of it.

Prerequisites #

• Prometheus 2.45+ with Alertmanager already receiving alerts and a working notification route (email, Slack, PagerDuty).
• The blackbox_exporter (0.24.0+) deployed reachable from Prometheus, and able to reach the portal’s public URL — probe from outside the Kubernetes cluster’s internal network so the probe crosses the same ingress and TLS a real client does.
• Defined SLOs for the endpoints, e.g. 99.9% availability and 95% of probes under 2s for the public WMS GetCapabilities.
• Access to the portal’s public OGC endpoint URLs and any required TLS settings.
• Permission to add scrape targets and rule files to Prometheus.

The flow below shows the prober issuing a real OGC request, Prometheus recording the probe result and burn rate, and a fast/slow burn-rate pair deciding whether Alertmanager pages.

Step-by-step implementation #

1. Define a blackbox probe module that checks a real OGC response #

A bare HTTP 200 is not enough — a portal can return 200 with an OGC ServiceException body, or a login page, and still look “up.” The probe module must assert on the response body. This module issues a GET, requires a 2xx status, verifies the TLS certificate, and matches the XML that only a healthy GetCapabilities returns:

modules:
  ogc_getcapabilities:
    prober: http
    timeout: 10s
    http:
      method: GET
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      fail_if_not_ssl: true
      # A healthy capabilities document contains this element; an
      # exception report or an error page will not, so the probe fails closed.
      fail_if_body_not_matches_regexp:
        - "<(wms:)?WMS_Capabilities"
      fail_if_body_matches_regexp:
        - "ServiceException"

2. Add Prometheus probe scrape targets #

Point Prometheus at the blackbox exporter with the target URLs as parameters. The relabel_configs idiom rewrites the scrape so __address__ becomes the exporter while the real endpoint travels as the target param and is preserved as the instance label.

scrape_configs:
  - job_name: blackbox-ogc
    metrics_path: /probe
    scrape_interval: 30s
    params:
      module: [ogc_getcapabilities]
    static_configs:
      - targets:
          - https://portal.example.gov/geoserver/wms?service=WMS&version=1.3.0&request=GetCapabilities
        labels:
          slo: wms-capabilities
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter.monitoring.svc:9115

3. Record availability and latency, then compute burn rate #

Recording rules pre-aggregate the raw probe_success and probe_duration_seconds into the ratios the alerts read, so the alert expressions stay simple and every dashboard agrees on one series. Availability is the mean of probe_success; the burn rate is the failure fraction divided by the error budget, evaluated over two windows.

groups:
  - name: ogc-slo-recording
    interval: 30s
    rules:
      - record: 'slo:probe_availability:ratio_rate5m'
        expr: 'avg by (slo) (avg_over_time(probe_success[5m]))'
      - record: 'slo:probe_availability:ratio_rate1h'
        expr: 'avg by (slo) (avg_over_time(probe_success[1h]))'
      # Failure fraction / error budget (0.001 for a 99.9% target) = burn rate.
      - record: 'slo:error_budget_burn:rate5m'
        expr: '(1 - slo:probe_availability:ratio_rate5m) / 0.001'
      - record: 'slo:error_budget_burn:rate1h'
        expr: '(1 - slo:probe_availability:ratio_rate1h) / 0.001'
      - record: 'slo:probe_latency:p95_5m'
        expr: 'histogram_quantile(0.95, sum by (slo, le) (rate(probe_http_duration_seconds_bucket[5m])))'

4. Fire a multi-window, multi-burn-rate alert #

A single-window alert either pages on every transient blip or reacts too slowly. The Google SRE multi-window pattern requires a fast window and a slow window to both exceed a threshold: a high burn rate confirmed over 5m and 1h means the budget is genuinely being consumed, not a one-off failed probe.

groups:
  - name: ogc-slo-alerts
    rules:
      - alert: OGCEndpointFastBurn
        expr: 'slo:error_budget_burn:rate5m > 14.4 and slo:error_budget_burn:rate1h > 14.4'
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Fast error-budget burn on {{ $labels.slo }}"
          description: "{{ $labels.slo }} is burning budget at {{ $value | printf \"%.1f\" }}x; the 30-day SLO is at risk within hours."
      - alert: OGCEndpointLatencySLO
        expr: 'slo:probe_latency:p95_5m > 2'
        for: 10m
        labels:
          severity: ticket
        annotations:
          summary: "p95 probe latency SLO breach on {{ $labels.slo }}"
          description: "95th-percentile probe latency is {{ $value | printf \"%.2f\" }}s, over the 2s objective."

5. Route the alert through Alertmanager #

Route by the severity label: page goes to the paging integration, ticket opens a lower-urgency notification. Group by slo so a single endpoint failing does not fan out into many pages.

route:
  receiver: default
  group_by: ["slo"]
  routes:
    - matchers: ['severity="page"']
      receiver: oncall-pager
      group_wait: 30s
      repeat_interval: 1h
    - matchers: ['severity="ticket"']
      receiver: ops-notifications
receivers:
  - name: default
    webhook_configs:
      - url: http://alert-sink.monitoring.svc:9099/
  - name: oncall-pager
    webhook_configs:
      - url: http://pager-bridge.monitoring.svc:8080/notify
  - name: ops-notifications
    webhook_configs:
      - url: http://ops-bridge.monitoring.svc:8080/notify

The endpoints being probed are the same ones fronted by the portal’s ingress, so pair this alerting with the caching and routing controls in Reverse Proxy Configuration for WMS/WFS — a probe that starts failing often points straight at a proxy change.

Verification #

# 1. The probe passes against a healthy endpoint (probe_success 1)
curl -s 'http://blackbox-exporter.monitoring.svc:9115/probe?module=ogc_getcapabilities&target=https://portal.example.gov/geoserver/wms?service=WMS%26request=GetCapabilities' \
  | grep '^probe_success'
#   expect: probe_success 1

# 2. Prometheus is scraping the blackbox job
curl -s 'http://prometheus.monitoring:9090/api/v1/query?query=probe_success{job="blackbox-ogc"}'

# 3. The burn-rate recording rule is populated
curl -s 'http://prometheus.monitoring:9090/api/v1/query?query=slo:error_budget_burn:rate5m'

# 4. Simulate a breach: point the target at a deliberately broken path and
#    confirm the alert enters Pending then Firing on the Prometheus /alerts page.

A probe_success 1 in step 1 with a matched body confirms the module asserts on content, not just status. If step 3 returns nothing, the recording rule file was not loaded — check rule_files in the Prometheus config and the /rules page.

Troubleshooting matrix #

Related #

• Monitoring and Observability for Geospatial Portals — the SLO and error-budget model these alerts implement.
• Exporting GeoServer Metrics to Prometheus — the internal request metrics that complement external probes.
• Reverse Proxy Configuration for WMS/WFS — the ingress a failing probe most often implicates.

Up one level: Monitoring and Observability for Geospatial Portals.