Geospatial Portal

Benchmarking Tile Latency with k6

A repeatable procedure for load-testing a map tile endpoint with k6, driving randomized z/x/y requests inside a bounding extent and reporting p95/p99 latency for cache-cold and cache-warm runs.

This is a hands-on companion to MapProxy vs TileServer GL for Tile Delivery and sits within the wider Infrastructure Orchestration & Configuration Management practice; read the parent guide first for why the two servers have different CPU and bandwidth profiles. Here the focus is narrow: how to measure real tail latency under load so a scaling threshold comes from data, not a vendor claim.

Prerequisites

  • k6 0.49+ installed (k6 version) — the Grafana load generator; the summary-export API used below is stable from 0.46 onward.
  • A reachable tile endpoint — either a MapProxy WMTS/TMS URL or a TileServer GL /data/<id>/{z}/{x}/{y}.pbf URL. Have both if you intend to compare them.
  • The z/x/y URL template and a bounding extent (min/max longitude and latitude) of a region that actually contains tiles, so requests are not all 404 misses.
  • Enough client bandwidth and CPU on the load-generator host that k6 itself is not the bottleneck; run it off the server under test.
  • Optional: a way to flush the tile cache between runs (restart MapProxy’s cache backend, or point at a freshly seeded archive) so cache-cold numbers are honest.

The harness below shows how the script turns a bounding extent into randomized tile requests, checks each response, and lets the thresholds decide the pass/fail exit code.

k6 tile-latency harness from staged virtual users to a threshold pass or fail Staged virtual users ramp from zero to a plateau and back down. Each virtual-user iteration picks a random zoom then a random x and y column and row inside the configured bounding extent, builds the tile URL, and sends an HTTP GET to the tile server under test. The response feeds a check for status 200 and a non-empty body, and its duration is recorded into the http_req_duration metric. When the run ends, the p95 and p99 thresholds plus the failure-rate threshold evaluate the collected metrics and set the process exit code to pass or fail. A summary JSON is exported alongside. stages (ramp VUs) 0 → plateau → 0 Random z/x/yinside bbox extent HTTP GET tileserver under test check() 200non-empty body http_req_durationrecorded per request request thresholds decide exit p95 < 400 ms p99 < 900 ms failure rate < 1% pass / fail summary.json exported alongside

Step-by-step implementation

1. Parameterize the endpoint and extent

Keep the base URL and tile-template out of the script body so the same file can hit MapProxy or TileServer GL without edits. k6 reads environment variables through __ENV. Decide your extent as a bounding box in longitude/latitude; the script converts lon/lat to tile columns and rows so requests land on real data.

# Compare the two servers by only swapping BASE_URL / TILE_PATH.
# MapProxy (TMS-style):
export BASE_URL="https://tiles.example.org"
export TILE_PATH="/tms/1.0.0/parcels/webmercator/{z}/{x}/{y}.png"

# TileServer GL (vector tiles) — same script, different env:
# export BASE_URL="https://vtiles.example.org"
# export TILE_PATH="/data/basemap/{z}/{x}/{y}.pbf"

export BBOX="-122.52,37.70,-122.35,37.83"   # minLon,minLat,maxLon,maxLat (San Francisco)
export MIN_ZOOM=10
export MAX_ZOOM=16

2. Write the k6 script

The script picks a random zoom, converts a random point in the bounding box to the x/y tile indices at that zoom (the standard slippy-map formula), requests the tile, and checks the response. Thresholds are declared in options so k6 sets a non-zero exit code when they fail — that is what makes the test usable as a CI gate.

import http from 'k6/http';
import { check } from 'k6';
import { Rate } from 'k6/metrics';

const tileErrors = new Rate('tile_errors');

const BASE_URL = __ENV.BASE_URL;
const TILE_PATH = __ENV.TILE_PATH;
const [minLon, minLat, maxLon, maxLat] = __ENV.BBOX.split(',').map(Number);
const MIN_ZOOM = Number(__ENV.MIN_ZOOM || 10);
const MAX_ZOOM = Number(__ENV.MAX_ZOOM || 16);

export const options = {
  // Ramp virtual users so the plateau is a steady load, not a spike.
  stages: [
    { duration: '30s', target: 20 },   // ramp up
    { duration: '2m', target: 20 },    // hold plateau
    { duration: '30s', target: 0 },    // ramp down
  ],
  thresholds: {
    // Fail the run (exit 99) if tail latency or errors exceed budget.
    http_req_duration: ['p(95)<400', 'p(99)<900'],
    tile_errors: ['rate<0.01'],
  },
};

function randInt(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

// Convert a lon/lat point to slippy-map tile x/y at a given zoom.
function lonLatToTile(lon, lat, z) {
  const n = Math.pow(2, z);
  const x = Math.floor(((lon + 180) / 360) * n);
  const latRad = (lat * Math.PI) / 180;
  const y = Math.floor(
    ((1 - Math.log(Math.tan(latRad) + 1 / Math.cos(latRad)) / Math.PI) / 2) * n
  );
  return { x, y };
}

export default function () {
  const z = randInt(MIN_ZOOM, MAX_ZOOM);
  const lon = minLon + Math.random() * (maxLon - minLon);
  const lat = minLat + Math.random() * (maxLat - minLat);
  const { x, y } = lonLatToTile(lon, lat, z);

  const path = TILE_PATH
    .replace('{z}', z)
    .replace('{x}', x)
    .replace('{y}', y);

  const res = http.get(`${BASE_URL}${path}`, {
    tags: { name: 'tile' }, // group all tiles under one metric row
  });

  const ok = check(res, {
    'status is 200': (r) => r.status === 200,
    'body is non-empty': (r) => r.body && r.body.length > 0,
  });
  tileErrors.add(!ok);
}

// Emit a machine-readable summary next to the console output.
export function handleSummary(data) {
  return {
    'summary.json': JSON.stringify(data, null, 2),
    stdout: '\n  see summary.json for full metrics\n',
  };
}

3. Run cache-cold, then cache-warm

Run the identical script twice against the same server. The first run, immediately after flushing the cache, measures the expensive path — reprojection and render-on-miss for MapProxy, cold OS file cache for TileServer GL. The second run, without flushing, measures steady-state warm delivery. The gap between the two is the single most important number for capacity planning.

# Cold run: flush the cache first, then test.
#   MapProxy: restart the cache backend or clear the cache dir/bucket.
k6 run --summary-export cold-summary.json tile-latency.js
mv summary.json cold-summary-full.json

# Warm run: do NOT flush — hit the now-populated cache.
k6 run --summary-export warm-summary.json tile-latency.js
mv summary.json warm-summary-full.json

4. Compare the two servers

Repeat step 3 against the other server by swapping only the two env vars from step 1. Keep every other variable — extent, zoom range, stages, thresholds, and load-generator host — identical, or the comparison is meaningless.

# MapProxy leg
export BASE_URL="https://tiles.example.org"
export TILE_PATH="/tms/1.0.0/parcels/webmercator/{z}/{x}/{y}.png"
k6 run --summary-export mapproxy-warm.json tile-latency.js

# TileServer GL leg (identical extent, zooms, stages)
export BASE_URL="https://vtiles.example.org"
export TILE_PATH="/data/basemap/{z}/{x}/{y}.pbf"
k6 run --summary-export tileserver-warm.json tile-latency.js

A benchmark that requests the same tile is not a benchmark

The single most common way a tile benchmark misleads is by having an unrealistic access pattern. Hammering one URL measures the cache and nothing else; requesting uniformly random tiles across the whole pyramid measures the renderer and nothing else. Real traffic is neither — it is heavily skewed towards a few popular areas and shallow zooms, with a long tail of everything else, and the hit ratio that results is the number that determines observed latency.

Access pattern decides what a tile benchmark measures Three columns — one tile, uniform random, and skewed — each with the hit ratio it produces, what it measures, and the error it introduces. One tile, repeated hit ratio: 100% measures: cache lookup and the network path error: reports a system that will look far slower in production useful only as a ceiling Uniform random hit ratio: near zero measures: the renderer at full cold-miss rate error: reports a system that will look far faster in production useful only as a floor Skewed, like real traffic hit ratio: the one you observe measures: the system as users experience it build it from a day of access logs, sampled and replayed the only one worth a threshold

Keep the first two patterns as bounding runs — they are genuinely useful for isolating a regression to the cache or the renderer — but never set a performance target from either. Derive the skewed pattern from real access logs, and regenerate it periodically, because the popular areas move when a new dataset is published or a story map is linked from a news site.

Verification

Confirm the run actually exercised the endpoint and that k6 honored the thresholds. A passing run exits 0; a threshold breach exits 99.

# 1. Threshold verdict from the process exit code (99 = a threshold failed).
k6 run tile-latency.js; echo "k6 exit code: $?"
#   k6 exit code: 0        <- all thresholds passed
#   k6 exit code: 99       <- p95/p99 or error-rate budget exceeded

# 2. Read the tail-latency percentiles straight out of the summary.
jq '.metrics.http_req_duration.values | {p95: .["p(95)"], p99: .["p(99)"]}' warm-summary.json
#   { "p95": 118.4, "p99": 233.7 }   <- warm run, milliseconds

# 3. Confirm requests actually succeeded (checks passed, low error rate).
jq '.metrics.tile_errors.values.rate, .root_group.checks' summary.json

# 4. Cold-vs-warm delta — the render-on-miss cost you are sizing for.
jq -n --argjson c "$(cat cold-summary.json)" --argjson w "$(cat warm-summary.json)" \
  '{cold_p95: $c.metrics.http_req_duration.values["p(95)"], warm_p95: $w.metrics.http_req_duration.values["p(95)"]}'

A large cold/warm gap points at expensive cache misses — the case for pre-seeding MapProxy hot extents. A small gap with high absolute latency points at the network or the load generator, not the server. Wire the exit code into a pipeline gate so a regression in tail latency blocks a release, the same discipline covered in Monitoring and Observability for Geospatial Portals.

Reporting a result that stays true next month

A benchmark number without its conditions is folklore, and tile benchmarks accumulate folklore quickly because so much affects them. Record the conditions alongside the number, in the same artefact, so that a future comparison is meaningful rather than an argument about whether the runs were alike.

The minimum set is small: the access pattern and its source, the cache state at the start of the run, the concurrency, the tile format and size, the deployed versions of the renderer and its spatial libraries, and whether the run went through the production edge or straight to the service. Cache state deserves particular care — a run against a warm cache and a run after an invalidation are measuring different systems, and it is the one condition most often left unstated.

The conditions that travel with the number Six recorded conditions grouped as workload, system and path, each with why it changes the result. Workload access pattern and its source concurrency and ramp tile format and dimensions changes which component is being measured at all System cache state at run start renderer version spatial library versions warm and cold runs are two different experiments Path through the production edge, or straight to the service client location and network bypassing the edge removes TLS, routing and rate limits

Report percentiles rather than averages, and report the error rate next to them: a p95 that improved because 4% of requests now fail fast is a regression wearing an improvement’s clothes. Finally, run long enough to include at least one of whatever happens periodically in the environment — a seeding job, a cache expiry wave, a backup — because a benchmark that only ever runs in the quiet minutes describes a system nobody uses.

Troubleshooting matrix

Symptom Likely cause Fix
Every request returns 404 Extent or zoom range has no tiles, or the URL template is wrong Verify BBOX covers real data; open one computed tile URL by hand; check TILE_PATH placeholders
p95 looks great but tile_errors is high check() passing on error pages, or 200s with empty bodies Assert body length and, for vector tiles, content-type application/x-protobuf
Latency dominated by the first few requests TLS handshake / connection setup counted in duration Compare http_req_waiting (server time) instead of total http_req_duration
Cold and warm runs look identical Cache was not actually flushed between runs Confirm the backend cleared; for TileServer GL drop OS page cache or restart the container
Numbers vary wildly between runs Load generator is the bottleneck, or noisy neighbor Run k6 off the server host; raise ulimit; watch generator CPU stays under ~70%
Threshold never fails despite slow tiles Thresholds too loose or metric mis-named Match the metric name exactly (http_req_duration); tighten p(95)/p(99) budgets
429 / connection resets under plateau Reverse-proxy rate limit tripped by the test Whitelist the generator IP or raise the limit for the test window

Up one level: MapProxy vs TileServer GL for Tile Delivery.