Tracing OGC Requests with OpenTelemetry

This guide adds distributed tracing to a geospatial portal so that a slow tile can be attributed to a tier rather than argued about, without the trace volume that tile traffic would otherwise produce. It belongs to Monitoring and Observability for Geospatial Portals, within the Infrastructure Orchestration & Configuration Management framework.

Prerequisites #

• An OpenTelemetry collector reachable from the cluster, with a trace backend behind it.
• A proxy able to generate and propagate trace context — Nginx with the OpenTelemetry module, or Envoy.
• Access to the renderer’s JVM arguments, so an agent can be attached.
• A metrics stack already in place, per exporting GeoServer metrics to Prometheus — tracing answers different questions and does not replace it.

What a trace answers that a metric cannot #

Metrics tell you that tile latency rose. A trace tells you where the time went in one specific request, which is the question that actually blocks an investigation. For a geospatial portal the answer is usually one of a small number of places, and they are indistinguishable from the outside.

Step-by-step implementation #

1. Start the trace at the edge #

# nginx.conf — generate trace context and propagate it upstream.
load_module modules/ngx_otel_module.so;

http {
    otel_exporter {
        endpoint otel-collector.observability.svc:4317;
    }
    otel_service_name geoportal-edge;

    # Head sampling here bounds cost; the tail decision happens in the collector.
    otel_trace on;
    otel_trace_context propagate;

    server {
        location /geoserver/ {
            otel_span_name "ogc $arg_request";
            otel_span_attr ogc.operation $arg_request;
            otel_span_attr ogc.layer     $arg_layers;
            otel_span_attr tile.z        $arg_z;
            otel_span_attr cache.status  $upstream_cache_status;
            proxy_pass http://geoserver_backend;
        }
    }
}

2. Attach an agent to the renderer #

# The Java agent instruments the servlet container and the JDBC driver, which is
# where the two spans that matter come from.
env:
  - name: JAVA_TOOL_OPTIONS
    value: "-javaagent:/otel/opentelemetry-javaagent.jar"
  - name: OTEL_SERVICE_NAME
    value: "geoserver"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector.observability.svc:4317"
  - name: OTEL_TRACES_SAMPLER
    value: "parentbased_always_on"      # honour the edge's decision, do not re-sample
  - name: OTEL_INSTRUMENTATION_JDBC_ENABLED
    value: "true"
  # Statement text can contain filter values; keep it sanitised.
  - name: OTEL_INSTRUMENTATION_COMMON_DB_STATEMENT_SANITIZER_ENABLED
    value: "true"

parentbased_always_on is the important line. If each tier samples independently, a request sampled at the edge may be unsampled at the renderer, producing a trace with a hole exactly where the interesting work happened.

3. Sample in a way tile volume can survive #

Tracing every tile request is neither affordable nor useful — the overwhelming majority are cache hits that complete in milliseconds and tell you nothing. The useful policy keeps a small baseline for shape and every trace that is slow or failed.

# collector config — keep what matters, discard the rest.
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-tiles
        type: latency
        latency: {threshold_ms: 500}
      - name: all-feature-queries       # rare and expensive: keep them all
        type: string_attribute
        string_attribute: {key: ogc.operation, values: [GetFeature, Transaction]}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 0.5}

4. Add the spans the default instrumentation misses #

// Two spans are worth adding by hand because they are where the time hides:
// waiting for a pooled connection, and reading from the tile store.
Span acquire = tracer.spanBuilder("pool.acquire")
    .setAttribute("pool.name", poolName)
    .startSpan();
try (Scope s = acquire.makeCurrent(); Connection c = pool.getConnection()) {
    acquire.setAttribute("pool.waited_ms", waitedMillis);
    // ... render
} finally {
    acquire.end();
}

Correlate traces, logs and metrics with one identifier #

A trace is most valuable when it can be reached from the other two signals. Propagate the trace id into the access log and into application log lines, so that a complaint with a timestamp becomes a trace in one query.

One caution about span attributes on a geospatial portal: bounding boxes, layer names and filter expressions are all tempting to record and all carry information about what somebody was looking at. Treat the attribute list as a deliberate allow-list rather than recording whatever the instrumentation offers, and review it with the same care as a log field — a trace backend is usually reachable by more people than the database is.

Verification #

# 1. A request produces a trace spanning both services
curl -s -o /dev/null -D - "https://portal.example.gov/geoserver/wms?service=WMS&request=GetMap&layers=parcels&bbox=-1.31,50.90,-1.28,50.92&width=512&height=512&format=image/png&srs=EPSG:4326" \
  | grep -i traceparent
#   then open that trace id in the backend: expect edge and renderer spans

# 2. Sampling is honoured downstream rather than re-decided
kubectl -n geoportal logs deploy/geoserver | grep -c 'sampled=false'
#   expect: no traces starting sampled at the edge and dropped at the renderer

# 3. The trace id appears in the access log
grep -o 'trace_id=[0-9a-f]\{32\}' /var/log/nginx/ogc_access.log | head -3

# 4. Slow requests are kept regardless of the baseline rate
#    Issue a deliberately slow feature query, then search the backend by duration.
#    expect: the trace present, with the database span visible

# 5. Trace volume is bounded
kubectl -n observability exec deploy/otel-collector -- \
  curl -s localhost:8888/metrics | grep otelcol_processor_tail_sampling_sampled
#   expect: a rate consistent with the policy, not with request volume

Troubleshooting matrix #

FAQ #

Does tracing replace the metrics stack? #

No, and treating it as a replacement is expensive. Metrics answer “is this happening and how often” cheaply across all traffic; traces answer “where did the time go in this one request” for a sampled subset. An alert should come from a metric, and the investigation that follows should land in a trace.

What sampling rate is right for a tile portal? #

A baseline well under one percent, plus every error and every request above a latency threshold, plus all feature and transaction operations. Tile hits are numerous and uninformative; feature queries are rare and expensive, and keeping all of them costs little while answering most of the questions that come up.

Can traces contain sensitive data? #

Yes, if the instrumentation records query parameters or statement text. A WFS filter can carry a person’s name or an address, and a span attribute travels to a backend with different access controls from the database. Enable statement sanitisation, allow-list the attributes recorded from request parameters, and treat the trace backend as a system holding portal data.

How much overhead does the agent add? #

For a JVM renderer, typically a few percent of CPU and a small fixed memory cost, and the export itself is asynchronous. The overhead that matters in practice is not the agent but the span volume: one span per tile at full rate is a serious load on the collector, which is what sampling exists to bound.

Should the database be traced too? #

The client side of the query is usually enough — the JDBC span shows how long the database took from the caller’s perspective, which is the number the portal cares about. Server-side instrumentation adds detail about planning and execution that is valuable during a specific investigation and rarely worth running continuously.

How do traces help with an incident rather than an investigation? #

They shorten the first step. During an incident the question is which tier to look at, and a handful of sampled traces from the affected window answers it in seconds rather than after three dashboards have been compared. Keep a saved query for “slow traces, last fifteen minutes, grouped by the largest span” — it is the fastest triage tool in the stack.

Should background jobs be traced as well as requests? #

Yes, and with a different sampling policy. An ingestion run is one logical operation made of thousands of tasks, and tracing every task is both expensive and unreadable. Trace the run as a parent span with a sampled subset of its tasks beneath, and propagate the run identifier into the task logs so the full detail remains reachable without every task producing spans.

Related #

• Exporting GeoServer Metrics to Prometheus — the signals that raise the alert.
• Aggregating Celery Logs with Loki — the third signal, joined by the same id.
• Alerting on OGC Endpoint SLA Breaches — what fires before anyone opens a trace.

Up one level: Monitoring and Observability for Geospatial Portals.