Geospatial Portal

Replicating Tile Caches Across Regions

This guide replicates a tile archive to a second region so that a regional failure does not require a full re-seed before maps work again. It belongs to Backup & Disaster Recovery for Spatial Platforms, within the Infrastructure Orchestration & Configuration Management framework.

Prerequisites

  • A tile archive in object storage rather than on a node’s local disk.
  • Two regions with storage in each, and a routing layer able to send traffic to either.
  • An invalidation mechanism that both regions observe — the publish event, not a manual purge.
  • A measured full-reseed duration, so the value of replication can be judged against it.

Replicate the cache, or re-render it?

Replication is not automatically worth it. The question is whether re-seeding in the second region is fast enough and cheap enough compared with keeping a continuously synchronised copy, and the answer depends on the archive’s size and on how quickly the portal must be fully useful after a failover.

Three levels of cross-region tile readiness Re-seed on failover, replicate shallow zooms only, and full replication, compared by continuous cost and by what a reader experiences immediately after failover. APPROACH CONTINUOUS COST AFTER FAILOVER re-seed on failover nothing stored twice zero slow maps for the whole seed duration shallow zooms only a few percent of the archive small and predictable orientation instantly, detail renders on demand full replication everything, both regions double storage plus transfer immediate parity

The middle row is the recommendation for most portals, and it is the same insight as the degraded-basemap approach applied across regions: the shallow levels are a tiny fraction of the tile count and carry most of the orientation value, while deep zoom is enormous and requested for a small area at a time.

Step-by-step implementation

1. Replicate the archive, filtered by zoom 

# Only shallow levels are replicated continuously. The prefix layout makes this
# a filter rather than a per-object decision.
aws s3api put-bucket-replication --bucket tiles-eu-west-2 \
  --replication-configuration '{
    "Role": "arn:aws:iam::123456789012:role/tile-replication",
    "Rules": [{
      "ID": "shallow-zooms",
      "Status": "Enabled",
      "Priority": 1,
      "Filter": {"Prefix": "basemap/"},
      "DeleteMarkerReplication": {"Status": "Enabled"},
      "Destination": {
        "Bucket": "arn:aws:s3:::tiles-eu-west-1",
        "StorageClass": "STANDARD"
      }
    }]
  }'

Replicating delete markers matters more than it looks: an invalidation in the primary region must remove the tile in the secondary as well, or a failover serves a version that was withdrawn.

2. Make invalidation reach both regions

The dangerous failure is not a missing tile — it is a stale one served confidently after a failover. Drive invalidation from the publish event so both regions act on the same signal.

# invalidate.py — publish an invalidation to a topic both regions subscribe to
import json, boto3

sns = boto3.client("sns", region_name="eu-west-2")

def invalidate(layer: str, bbox: tuple[float, float, float, float], zmax: int = 11):
    """Announce that a layer's tiles are superseded, to every region."""
    sns.publish(
        TopicArn="arn:aws:sns:eu-west-2:123456789012:tile-invalidation",
        Message=json.dumps({
            "layer": layer,
            "bbox": list(bbox),
            "zoom_max": zmax,          # only the replicated levels need purging
            "version": current_version(layer),
        }),
        MessageAttributes={"layer": {"DataType": "String", "StringValue": layer}},
    )
Invalidation must travel faster than replication Two propagation paths — object replication and event-driven invalidation — with the window during which a stale tile could be served if only the first existed. A new tile version is published in the primary region object replication — minutes, and variable invalidation event — seconds Why the order matters If the secondary learned of the change only when the object arrived, it would keep serving the superseded tile for the whole replication lag — and would do so most visibly during a failover, when nobody is watching that region. Purge on the event; let replication refill at its own pace. A missing tile renders on demand; a wrong tile misleads.

3. Route deliberately, and only one way at a time 

# The secondary region serves only when the primary is unhealthy, and announces
# itself so a stale-tile report can be attributed to a region.
upstream tiles_primary   { server tiles-eu-west-2.internal:8080 max_fails=3 fail_timeout=20s; }
upstream tiles_secondary { server tiles-eu-west-1.internal:8080 backup; }

server {
    location ~ ^/wmts/ {
        proxy_pass http://tiles_primary;
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
        add_header X-Tile-Region $upstream_addr always;
    }
}

4. Decide what happens to writes during a regional failover

Serving tiles from a second region is straightforward; accepting publishes there is not. A portal that fails over reads but not writes is a coherent, defensible state — a portal that accepts publishes in both regions has created a divergence that somebody must reconcile by hand.

Three postures for a regional failover Reads-only, promoted single writer, and active-active, compared by what each requires and what it risks. reads only one write region, always publishes pause during failover simple, and correct by default the right posture for a short regional outage promoted writer the database is promoted the old primary must be fenced a deliberate, recorded decision for outages measured in days rather than hours active-active writes both regions accept publishes conflicts need resolution spatial edits rarely merge cleanly avoid unless there is a real requirement and an owner

Verification 

# 1. Replication is actually running and current
aws s3api head-object --bucket tiles-eu-west-1 --key basemap/6/32/21.png \
  --query 'ReplicationStatus'
#   expect: "REPLICA"

# 2. Only the intended zoom levels are replicated
aws s3 ls --recursive s3://tiles-eu-west-1/basemap/ | awk -F/ '{print $2}' | sort -u
#   expect: 0 through 11

# 3. An invalidation removes the tile in BOTH regions
python3 -c "import invalidate; invalidate.invalidate('basemap', (-1.4,50.8,-1.2,51.0))"
for R in eu-west-2 eu-west-1; do
  aws s3api head-object --bucket tiles-$R --key basemap/11/1010/680.png >/dev/null 2>&1 \
    && echo "$R: still present" || echo "$R: purged"
done
#   expect: purged in both, within seconds

# 4. Failover serves from the secondary, and says so
sudo systemctl stop tile-cache        # on the primary
curl -sI "https://tiles.example.gov/wmts/basemap/6/32/21.png" | grep -E 'HTTP/|X-Tile-Region'
#   expect: 200, and a region header naming the secondary

# 5. Deep zoom in the secondary renders rather than 404s
curl -s -o /dev/null -w '%{http_code}\n' "https://tiles.example.gov/wmts/basemap/15/16100/10900.png"
#   expect: 200 — rendered on demand, slower, but correct

Testing the secondary before you need it

A replicated archive that has never served a request is an assumption. The failure modes are unglamorous and entirely predictable: a bucket policy that permits replication but not reads, a routing rule that was never applied in that region, a certificate that was issued for the primary hostname only, a renderer image that was never pulled into the secondary’s registry.

The cheapest continuous test is to route a small, deliberate share of real traffic to the secondary — a percentage of anonymous tile requests, or all traffic from an internal test client — so that the path is exercised every day rather than once a quarter. This is different from a health check: a health check confirms that something answers, while real traffic confirms that the right thing is served, with the right headers, at an acceptable latency, through the real routing layer.

Where a traffic split is not acceptable, the next best option is a scheduled synthetic exercise: force a failover for a fixed window at a quiet hour, confirm the criteria, and fail back. Fifteen minutes a month is enough to catch every one of the failure modes above, and each of them is the kind that otherwise surfaces during the incident it was meant to mitigate.

Whichever approach is used, assert on content rather than on status. A secondary region that returns HTTP 200 with a tile from a superseded version has passed every naive check and failed at the only thing that matters — so compare a hash of a known tile against the primary’s, and alert on divergence rather than on availability alone.

Troubleshooting matrix

Symptom Likely cause Fix
The secondary serves a superseded tile after failover Invalidation travelled by replication rather than by event Publish invalidations to a topic both regions consume
Replication lag grows without bound Deep zooms included in the replication filter Restrict the filter to shallow levels; let deep zoom render on demand
Deletes in the primary do not remove tiles in the secondary Delete-marker replication disabled Enable it, and verify with a purge test rather than assuming
Both regions serve simultaneously with different content Routing weighted rather than failover-only Make the secondary a backup peer, and fail over deliberately
Failover works but publishes fail confusingly Writes attempted in a region with a read-only replica Decide the posture in advance; pause publishing rather than half-accepting it
Cross-region transfer cost is higher than expected Replicating tiles that are never requested in the secondary Replicate by request pattern, not by completeness
Nobody knows which region served a report of a bad tile No region marker on responses Emit a region header and include it in support reports

FAQ

Should the secondary region run a renderer as well?

Yes, if deep zoom must work there. A replicated shallow archive plus a renderer gives a fully functional, slower portal; a replicated archive alone gives orientation and 404s below the replicated levels, which is acceptable for a short outage and not for a long one.

How is this different from a backup?

It is availability rather than recovery. Replication faithfully copies whatever the primary holds, including a bad publish, within minutes. It does not let you go back to yesterday, which is what backups are for — the distinction is set out in the parent topic.

Does the tile cache need to be consistent with the database across regions?

It needs to be no more current than the database in that region, which is the condition failover breaks if the cache replicates faster than the data. Where both are replicated, publish invalidations from the same event that publishes the data, so the two move together.

What about vector tiles?

The same structure, with a smaller archive and one extra consideration: the style is a separate artefact and must be replicated with the tiles, or the secondary renders correct geometry with the wrong cartography — which is harder to notice than a missing tile.

How should the secondary’s cost be kept from growing quietly?

Review what is actually being replicated against what is actually requested there, quarterly. Replication rules are written once, prefixes change as new layers are published, and the usual outcome is that a layer nobody consults in the secondary region is being copied continuously for years. Reporting replicated bytes per layer alongside requests served per layer makes the mismatch obvious in one glance, and the fix is a filter change rather than a project.

Up one level: Backup & Disaster Recovery for Spatial Platforms.