Geospatial Portal

Tuning MapProxy Cache Seeding for Large Extents

How to seed a MapProxy tile cache across national or continental extents without blowing out disk, saturating the source WMS, or leaving half-rendered zoom levels behind.

This walkthrough is a hands-on companion to GeoNode vs MapProxy Architecture Comparison and sits within the wider Core Portal Architecture & Security Boundaries practice; read the parent comparison first if you are still deciding whether MapProxy or a bundled GeoNode cache owns your tile pipeline. Here the scope is one operational task: writing a seed.yaml that pre-renders exactly the tiles a large-extent basemap needs and nothing more.

Prerequisites

Naive seeding of a wide extent to deep zoom levels can mean billions of tiles. Bound the job before you run it.

  • MapProxy 1.16+ with the mapproxy-seed CLI on the path and a working mapproxy.yaml whose caches and grids you intend to seed.
  • A reachable source WMS or WMTS that can sustain the concurrency you plan to point at it; confirm you are permitted to hammer it.
  • A coverage geometry: a bounding box, or a polygon file (GeoJSON, Shapefile, or a PostGIS query) in a known SRS to clip seeding to land, a country, or a service area.
  • Disk headroom sized from a tile-count estimate, not a guess — a web-mercator grid roughly quadruples tile count per zoom level.
  • A writable cache backend (file, MBTiles, or S3) and permission to run long-lived jobs off-peak.
  • gdalinfo/ogrinfo or mapproxy-util handy to sanity-check SRS and extents before committing to a multi-hour run.

The diagram below is the mental model for bounding a seed job: a tile pyramid whose per-level tile count explodes with depth, clipped by a coverage polygon so only tiles intersecting the area of interest are rendered.

Coverage-clipped tile pyramid for large-extent seeding A pyramid of zoom levels is drawn top to bottom. Level zero is a single tile; each level down doubles in each dimension so the tile count roughly quadruples, reaching a very wide base at deep zoom. A shaded coverage polygon overlays the pyramid, and only the portion of each level that falls inside the polygon is marked for seeding. A callout notes that constraining the level range and the coverage together is what keeps the job from producing billions of tiles at the base. z0 · 1 tile z1–z3 · tens z4–z6 · thousands z7–z9 · millions z10+ · billions without a coverage coverage clip level range + coverage bound the base

Step-by-step implementation

1. Define coverages in seed.yaml

Every seed job begins with a named coverage that constrains where tiles are rendered. Use a bounding box for a rectangular region, or a geometry file when you must clip to a coastline or administrative boundary. Always declare the coverage SRS — a mismatched SRS silently seeds the wrong area.

coverages:
  mainland_bbox:
    bbox: [-124.8, 24.4, -66.9, 49.4]
    srs: 'EPSG:4326'
  service_area_poly:
    datasource: '/data/coverages/service_area.geojson'
    srs: 'EPSG:4326'
    # only tiles intersecting this polygon are seeded

A polygon coverage is what turns a rectangular billions-of-tiles job into a tractable one: seeding a coastal service area clipped to land can eliminate the majority of ocean tiles a bounding box would otherwise render. Point datasource at any OGR-readable file, or at a PostGIS table if your boundaries already live in the database.

2. Constrain the level range per coverage

The second lever is depth. A single seeds entry ties a cache to a coverage and a levels range; keep low zooms global and reserve deep zooms for the tight coverage so tile count stays bounded.

seeds:
  overview_levels:
    caches: [basemap_cache]
    grids: [webmercator]
    coverages: [mainland_bbox]
    levels:
      from: 0
      to: 8
  detail_levels:
    caches: [basemap_cache]
    grids: [webmercator]
    coverages: [service_area_poly]
    levels:
      from: 9
      to: 16

Splitting the job in two — a shallow range over the wide box and a deep range over the clipped polygon — is the core tactic for large extents: you get a complete low-zoom basemap everywhere and high-resolution tiles only where users actually pan. This mirrors the layering trade-off discussed in the parent architecture comparison.

3. Set concurrency against the source’s ceiling

mapproxy-seed -c sets how many tiles render in parallel. Too low wastes hours; too high overruns the source WMS and trips its rate limits. Tune to the upstream’s capacity, not the seeder’s.

# Dry-run first: report the tile count WITHOUT rendering
mapproxy-seed -f mapproxy.yaml -s seed.yaml --dry-run --summary

# Then seed with bounded concurrency (8 parallel render workers)
mapproxy-seed -f mapproxy.yaml -s seed.yaml \
  --seed overview_levels --concurrency 8

Always run --dry-run --summary first: it prints the tile count and estimated size per seed so you can catch a runaway job before it writes a single tile. Raise --concurrency only until the source WMS response time starts climbing, then back off one step.

4. Run incremental seeding with refresh_before

For updates, you rarely want to re-render everything. refresh_before re-seeds only tiles older than a cutoff, so a nightly job touches just what has changed while leaving warm tiles alone.

seeds:
  nightly_refresh:
    caches: [basemap_cache]
    grids: [webmercator]
    coverages: [service_area_poly]
    levels:
      from: 9
      to: 16
    refresh_before:
      weeks: 1

With refresh_before: {weeks: 1}, the seeder skips any tile rendered within the last week and regenerates the rest — turning a full multi-hour reseed into a short incremental one. Pair this with a cleanup section to purge tiles that fall outside a shrunken coverage so stale bytes do not accumulate.

5. Monitor progress and disk during the run

A large seed is a long-running job; watch both its render progress and the filling disk so you can abort before the volume fills.

# Verbose progress: percent complete, ETA, tiles/second
mapproxy-seed -f mapproxy.yaml -s seed.yaml \
  --seed detail_levels --concurrency 8 --progress-file .seed_progress -v

# In another shell, watch cache growth against free space
watch -n 30 'du -sh /data/mapproxy/tiles; df -h /data/mapproxy'

The --progress-file lets an interrupted job resume where it stopped instead of restarting from zoom zero — essential when a national seed spans hours and a node may be recycled mid-run. If the df free space trends toward zero faster than the ETA, stop and tighten the coverage or level range.

6. Schedule off-peak reseeds

Fold the incremental refresh into a scheduler so it runs when both the source WMS and the portal are quiet, and so an overrun never collides with peak read traffic.

# /etc/cron.d/mapproxy-seed  — run the incremental refresh at 02:15 nightly
15 2 * * *  mapproxy  /usr/local/bin/mapproxy-seed -f /etc/mapproxy/mapproxy.yaml -s /etc/mapproxy/seed.yaml --seed nightly_refresh --concurrency 6 >> /var/log/mapproxy/seed.log 2>&1

Scheduling the deep-zoom refresh for the small hours keeps render load off the source during business hours and means the tiles your stale-while-revalidate layer serves are already warm by morning — the freshness technique in Configuring Stale-While-Revalidate Tile Caching.

Verification

Confirm the seed produced the tiles you intended and no more.

# 1. Dry-run count matches expectation before a real run
mapproxy-seed -f mapproxy.yaml -s seed.yaml --dry-run --summary
#   expect: a tile total in the millions, not billions

# 2. Deep-zoom tiles exist only inside the coverage polygon
find /data/mapproxy/tiles/basemap_cache/14 -name '*.png' | wc -l
#   expect: non-zero, but far below a full-level count

# 3. A sampled tile actually renders (not a 0-byte file)
gdalinfo /data/mapproxy/tiles/basemap_cache/14/EPSG3857/00/... .png >/dev/null && echo OK

# 4. Cache size tracks the estimate from the dry run
du -sh /data/mapproxy/tiles

A dry-run count that lands in the expected order of magnitude, deep-zoom tiles present only within the coverage, and a cache size close to the estimate together prove the job was correctly bounded. Wire the dry-run count into CI so a coverage or level change that would 10x the tile count is caught before it runs.

Troubleshooting matrix

Symptom Likely cause Fix
Seed job estimates billions of tiles Deep levels.to applied over a wide bbox instead of a clipped polygon Split into shallow-box and deep-polygon seeds; add a coverages clip to the deep range
Source WMS returns errors or throttles mid-run --concurrency exceeds the upstream’s capacity Lower concurrency until source response time stabilizes, then resume via the progress file
Deep-zoom tiles rendered over ocean/empty areas Coverage is a bbox, not a land/service polygon Point the coverage datasource at a GeoJSON/PostGIS boundary and set its SRS
Nightly refresh re-renders everything refresh_before omitted, so all tiles are regenerated Add refresh_before with a week/day cutoff to touch only aged tiles
Disk fills before the job finishes Tile count under-estimated; no dry run performed Run --dry-run --summary first; tighten level range or coverage; add a cleanup section
Interrupted job restarts from zoom 0 No --progress-file, so no resume state Always pass --progress-file; resume the same job after a node recycle
Tiles seeded in the wrong area Coverage SRS mismatched with the grid SRS Set the coverage srs explicitly and confirm it against the grid definition

For how a standalone MapProxy seed pipeline compares with GeoNode’s bundled tile handling — and when to run each — see the parent GeoNode vs MapProxy Architecture Comparison.

Up one level: GeoNode vs MapProxy Architecture Comparison.