Geospatial Portal

Tuning MapProxy Cache Seeding for Large Extents

How to seed a MapProxy tile cache across national or continental extents without blowing out disk, saturating the source WMS, or leaving half-rendered zoom levels behind.

This walkthrough is a hands-on companion to GeoNode vs MapProxy Architecture Comparison and sits within the wider Core Portal Architecture & Security Boundaries practice; read the parent comparison first if you are still deciding whether MapProxy or a bundled GeoNode cache owns your tile pipeline. Here the scope is one operational task: writing a seed.yaml that pre-renders exactly the tiles a large-extent basemap needs and nothing more.

Prerequisites

Naive seeding of a wide extent to deep zoom levels can mean billions of tiles. Bound the job before you run it.

  • MapProxy 1.16+ with the mapproxy-seed CLI on the path and a working mapproxy.yaml whose caches and grids you intend to seed.
  • A reachable source WMS or WMTS that can sustain the concurrency you plan to point at it; confirm you are permitted to hammer it.
  • A coverage geometry: a bounding box, or a polygon file (GeoJSON, Shapefile, or a PostGIS query) in a known SRS to clip seeding to land, a country, or a service area.
  • Disk headroom sized from a tile-count estimate, not a guess — a web-mercator grid roughly quadruples tile count per zoom level.
  • A writable cache backend (file, MBTiles, or S3) and permission to run long-lived jobs off-peak.
  • gdalinfo/ogrinfo or mapproxy-util handy to sanity-check SRS and extents before committing to a multi-hour run.

The diagram below is the mental model for bounding a seed job: a tile pyramid whose per-level tile count explodes with depth, clipped by a coverage polygon so only tiles intersecting the area of interest are rendered.

Coverage-clipped tile pyramid for large-extent seeding A pyramid of zoom levels is drawn top to bottom. Level zero is a single tile; each level down doubles in each dimension so the tile count roughly quadruples, reaching a very wide base at deep zoom. A shaded coverage polygon overlays the pyramid, and only the portion of each level that falls inside the polygon is marked for seeding. A callout notes that constraining the level range and the coverage together is what keeps the job from producing billions of tiles at the base. z0 · 1 tile z1–z3 · tens z4–z6 · thousands z7–z9 · millions z10+ · billions without a coverage coverage clip level range + coverage bound the base

Step-by-step implementation

1. Define coverages in seed.yaml

Every seed job begins with a named coverage that constrains where tiles are rendered. Use a bounding box for a rectangular region, or a geometry file when you must clip to a coastline or administrative boundary. Always declare the coverage SRS — a mismatched SRS silently seeds the wrong area.

coverages:
  mainland_bbox:
    bbox: [-124.8, 24.4, -66.9, 49.4]
    srs: 'EPSG:4326'
  service_area_poly:
    datasource: '/data/coverages/service_area.geojson'
    srs: 'EPSG:4326'
    # only tiles intersecting this polygon are seeded

A polygon coverage is what turns a rectangular billions-of-tiles job into a tractable one: seeding a coastal service area clipped to land can eliminate the majority of ocean tiles a bounding box would otherwise render. Point datasource at any OGR-readable file, or at a PostGIS table if your boundaries already live in the database.

2. Constrain the level range per coverage

The second lever is depth. A single seeds entry ties a cache to a coverage and a levels range; keep low zooms global and reserve deep zooms for the tight coverage so tile count stays bounded.

seeds:
  overview_levels:
    caches: [basemap_cache]
    grids: [webmercator]
    coverages: [mainland_bbox]
    levels:
      from: 0
      to: 8
  detail_levels:
    caches: [basemap_cache]
    grids: [webmercator]
    coverages: [service_area_poly]
    levels:
      from: 9
      to: 16

Splitting the job in two — a shallow range over the wide box and a deep range over the clipped polygon — is the core tactic for large extents: you get a complete low-zoom basemap everywhere and high-resolution tiles only where users actually pan. This mirrors the layering trade-off discussed in the parent architecture comparison.

3. Set concurrency against the source’s ceiling

mapproxy-seed -c sets how many tiles render in parallel. Too low wastes hours; too high overruns the source WMS and trips its rate limits. Tune to the upstream’s capacity, not the seeder’s.

# Dry-run first: report the tile count WITHOUT rendering
mapproxy-seed -f mapproxy.yaml -s seed.yaml --dry-run --summary

# Then seed with bounded concurrency (8 parallel render workers)
mapproxy-seed -f mapproxy.yaml -s seed.yaml \
  --seed overview_levels --concurrency 8

Always run --dry-run --summary first: it prints the tile count and estimated size per seed so you can catch a runaway job before it writes a single tile. Raise --concurrency only until the source WMS response time starts climbing, then back off one step.

4. Run incremental seeding with refresh_before

For updates, you rarely want to re-render everything. refresh_before re-seeds only tiles older than a cutoff, so a nightly job touches just what has changed while leaving warm tiles alone.

seeds:
  nightly_refresh:
    caches: [basemap_cache]
    grids: [webmercator]
    coverages: [service_area_poly]
    levels:
      from: 9
      to: 16
    refresh_before:
      weeks: 1

With refresh_before: {weeks: 1}, the seeder skips any tile rendered within the last week and regenerates the rest — turning a full multi-hour reseed into a short incremental one. Pair this with a cleanup section to purge tiles that fall outside a shrunken coverage so stale bytes do not accumulate.

5. Monitor progress and disk during the run

A large seed is a long-running job; watch both its render progress and the filling disk so you can abort before the volume fills.

# Verbose progress: percent complete, ETA, tiles/second
mapproxy-seed -f mapproxy.yaml -s seed.yaml \
  --seed detail_levels --concurrency 8 --progress-file .seed_progress -v

# In another shell, watch cache growth against free space
watch -n 30 'du -sh /data/mapproxy/tiles; df -h /data/mapproxy'

The --progress-file lets an interrupted job resume where it stopped instead of restarting from zoom zero — essential when a national seed spans hours and a node may be recycled mid-run. If the df free space trends toward zero faster than the ETA, stop and tighten the coverage or level range.

6. Schedule off-peak reseeds

Fold the incremental refresh into a scheduler so it runs when both the source WMS and the portal are quiet, and so an overrun never collides with peak read traffic.

# /etc/cron.d/mapproxy-seed  — run the incremental refresh at 02:15 nightly
15 2 * * *  mapproxy  /usr/local/bin/mapproxy-seed -f /etc/mapproxy/mapproxy.yaml -s /etc/mapproxy/seed.yaml --seed nightly_refresh --concurrency 6 >> /var/log/mapproxy/seed.log 2>&1

Scheduling the deep-zoom refresh for the small hours keeps render load off the source during business hours and means the tiles your stale-while-revalidate layer serves are already warm by morning — the freshness technique in Configuring Stale-While-Revalidate Tile Caching.

Budgeting a seed run before you start it

A seed run is an expensive, long-lived batch job, and the three resources it consumes — renderer CPU, cache storage, and wall-clock time — are all decided before the first tile is drawn, by the levels and coverage you chose. Estimating them takes a minute and routinely changes the plan, because the tile count roughly quadruples per zoom level: the cost of levels 0–14 is a rounding error next to level 18, and a run that looks reasonable at the top of the pyramid is unfinishable at the bottom.

Where a seed run's cost actually sits: relative tile count per zoom level Bars for zoom levels 12 through 18 over one coverage area. Each bar is about four times the previous, so level 18 alone is larger than every shallower level combined. Tiles per level over one coverage polygon (relative, log-free view) z12 z13 z14 z15 z16 z17 z18 tile count the last two levels are the entire budget — cut coverage there first, not everywhere

Turn that shape into three numbers before scheduling anything. Multiply the dry-run tile count by the average rendered tile size to get storage, and check it against free space with headroom for the cache’s own metadata. Divide the tile count by the measured throughput of a short sample run — tiles per second at the concurrency you intend to use — to get wall-clock time, then ask whether that fits in the maintenance window you actually have. If either number is uncomfortable, the fix is almost always to tighten the coverage polygon at the deepest two levels rather than to drop a level entirely, because deep tiles outside populated areas are the ones nobody requests.

Verification

Confirm the seed produced the tiles you intended and no more.

# 1. Dry-run count matches expectation before a real run
mapproxy-seed -f mapproxy.yaml -s seed.yaml --dry-run --summary
#   expect: a tile total in the millions, not billions

# 2. Deep-zoom tiles exist only inside the coverage polygon
find /data/mapproxy/tiles/basemap_cache/14 -name '*.png' | wc -l
#   expect: non-zero, but far below a full-level count

# 3. A sampled tile actually renders (not a 0-byte file)
gdalinfo /data/mapproxy/tiles/basemap_cache/14/EPSG3857/00/... .png >/dev/null && echo OK

# 4. Cache size tracks the estimate from the dry run
du -sh /data/mapproxy/tiles

A dry-run count that lands in the expected order of magnitude, deep-zoom tiles present only within the coverage, and a cache size close to the estimate together prove the job was correctly bounded. Wire the dry-run count into CI so a coverage or level change that would 10x the tile count is caught before it runs.

Keeping the seed off the readers’ path

The failure that turns a routine refresh into an incident is resource contention: the seeding workers and live readers share one renderer and one database, and a seed run at full concurrency is a sustained load spike shaped exactly like a traffic surge. Readers experience it as slow tiles, the tile proxy experiences it as timeouts, and the failover logic described in fallback routing strategies for tile servers may start diverting traffic to a tier that is competing for the same CPU.

Three ways to keep a seed run from competing with live tile traffic Three side-by-side options — separate in time, separate in capacity, separate by rate — each with when to use it and what it costs. Separate in time seed at 02:00 with full concurrency, stop by 05:00 use when: traffic has a real nightly trough costs: the run must finish inside the window, or it is killed mid-level Separate in capacity seed-only renderer pods, writing to the shared cache use when: traffic is round-the- clock, or the run takes days costs: extra compute, and the database is still shared — isolate its pool too Separate by rate run continuously at low concurrency, never stopping use when: the cache is refreshed incrementally, not rebuilt costs: hardest to reason about — spare capacity moves, so the rate must adapt or be conservative

Whichever you pick, make the seeding load observable as its own series rather than mixed into total renderer load. Tag seed requests with a distinct user agent or header at the proxy so the metrics split cleanly, and you can answer the only question that matters during a slow-tiles complaint: is the portal busy because readers arrived, or because a refresh is running?

Troubleshooting matrix

Symptom Likely cause Fix
Seed job estimates billions of tiles Deep levels.to applied over a wide bbox instead of a clipped polygon Split into shallow-box and deep-polygon seeds; add a coverages clip to the deep range
Source WMS returns errors or throttles mid-run --concurrency exceeds the upstream’s capacity Lower concurrency until source response time stabilizes, then resume via the progress file
Deep-zoom tiles rendered over ocean/empty areas Coverage is a bbox, not a land/service polygon Point the coverage datasource at a GeoJSON/PostGIS boundary and set its SRS
Nightly refresh re-renders everything refresh_before omitted, so all tiles are regenerated Add refresh_before with a week/day cutoff to touch only aged tiles
Disk fills before the job finishes Tile count under-estimated; no dry run performed Run --dry-run --summary first; tighten level range or coverage; add a cleanup section
Interrupted job restarts from zoom 0 No --progress-file, so no resume state Always pass --progress-file; resume the same job after a node recycle
Tiles seeded in the wrong area Coverage SRS mismatched with the grid SRS Set the coverage srs explicitly and confirm it against the grid definition

For how a standalone MapProxy seed pipeline compares with GeoNode’s bundled tile handling — and when to run each — see the parent GeoNode vs MapProxy Architecture Comparison.

Up one level: GeoNode vs MapProxy Architecture Comparison.