Geospatial Portal

Forecasting Tile Storage Growth

This guide turns tile-archive growth from a surprise on a bill into arithmetic anyone can do before a cartography decision is made. It belongs to Autoscaling & Capacity Planning for Geospatial Workloads, within the Infrastructure Orchestration & Configuration Management framework.

Prerequisites

  • An existing tile archive with per-level object counts and sizes available from the storage layer.
  • The coverage polygon and zoom range each layer is seeded to, from the seeding configuration.
  • A month of access logs, to distinguish tiles that are requested from tiles that merely exist.
  • A cost per stored gigabyte for each storage class in use.

The arithmetic that decides everything

Tile counts quadruple per zoom level. That single fact is the whole forecast, and it means the last level in a range costs more than every level above it combined — so a decision to add “one more zoom level” is not an incremental change.

Each zoom level costs more than everything above it Bars for zoom levels 10 to 17 growing fourfold each, with a note that adding one level roughly doubles the archive. Tiles per level over one coverage polygon z10 z11 z12 z13 z14 z15 z16 z17 Adding one level roughly doubles the archive; adding two roughly quintuples it. This is arithmetic, not an estimate.

Step-by-step implementation

1. Measure what you actually have, per level and per style 

# Object count and bytes per zoom level, from the storage layer rather than
# from the seeding configuration — they diverge.
aws s3 ls --recursive s3://portal-tiles/basemap/ \
  | awk '{split($4,p,"/"); size[p[3]] += $3; count[p[3]]++}
         END {for (z in size) printf "z%-4s %10d tiles %12.2f GB\n", z, count[z], size[z]/1073741824}' \
  | sort -V
# And per style, because a second style is a second archive.
for STYLE in default print accessible; do
  echo -n "$STYLE: "
  aws s3 ls --summarize --recursive "s3://portal-tiles/$STYLE/" | awk '/Total Size/ {printf "%.1f GB\n", $3/1073741824}'
done

2. Forecast a change before making it 

# tile_forecast.py — what will this change cost, before we make it?
AVG_TILE_BYTES = 14_500          # measured from the existing archive, per style

def tiles_at_level(area_fraction: float, z: int) -> int:
    """Tiles intersecting a coverage that is `area_fraction` of the world at zoom z."""
    return int(area_fraction * (4 ** z))

def forecast(area_fraction: float, z_from: int, z_to: int, styles: int = 1) -> float:
    total = sum(tiles_at_level(area_fraction, z) for z in range(z_from, z_to + 1))
    return total * AVG_TILE_BYTES * styles / 1_073_741_824      # GB

# A national extent is a small fraction of the world; measure yours rather than
# assuming, because coastline and coverage shape matter more than area alone.
AREA = 0.00006

for zmax in (14, 15, 16, 17):
    print(f"z0-z{zmax}, 1 style: {forecast(AREA, 0, zmax):9.1f} GB")
print(f"z0-z16, 3 styles:  {forecast(AREA, 0, 16, styles=3):9.1f} GB")

3. Separate what is requested from what merely exists

How much of the archive is ever read Stored versus requested tiles at shallow, mid and deep zoom levels, showing the divergence at depth. Stored (outline) versus requested (filled), per level band shallow z0-z10 nearly all of it is read — seed it mid z11-z14 most of it — seed the populated parts deep z15+ a small fraction — render on demand and cache the result Seeding the deepest level across the whole coverage stores a large volume nobody ever asks for. Let readers create it. 
# Which stored levels are actually requested? Compare the two directly.
awk '{match($7, /\/([0-9]+)\/[0-9]+\/[0-9]+\.png/, m); if (m[1] != "") req[m[1]]++}
     END {for (z in req) printf "z%-4s %10d requests\n", z, req[z]}' \
  /var/log/nginx/tiles.log | sort -V

4. Apply lifecycle rules to the tail 

# Deep-zoom tiles that have not been read in 90 days are a rebuild candidate,
# not a permanent asset. Move them to cheaper storage, or expire them entirely
# and let the renderer recreate them on demand.
aws s3api put-bucket-lifecycle-configuration --bucket portal-tiles \
  --lifecycle-configuration '{
    "Rules": [
      {"ID": "deep-zoom-cooldown", "Status": "Enabled",
       "Filter": {"Prefix": "basemap/16/"},
       "Transitions": [{"Days": 90, "StorageClass": "STANDARD_IA"}]},
      {"ID": "deep-zoom-expiry", "Status": "Enabled",
       "Filter": {"Prefix": "basemap/17/"},
       "Expiration": {"Days": 180}}
    ]}'

Expiring deep tiles is safe when the renderer can recreate them and the fallback behaviour is understood; it is not safe when the renderer has been retired on the assumption that the archive is complete. Decide which of those is true before adding the rule.

Put the forecast where the decision is made

A forecast in a spreadsheet nobody opens changes nothing. The value comes from putting the number in front of the person choosing a zoom range or adding a style, at the moment they choose it.

Where a forecast has to live to matter Spreadsheet, quarterly review and pull-request check, compared by when the number is seen relative to the decision. a spreadsheet read once, when it was written no influence a quarterly review explains growth after it has happened some influence a check on the change posts the forecast on the pull request that edits the seeding config changes decisions before the decision is final, in front of the person making it

The check is a few lines: parse the zoom range and coverage from the changed configuration, run the forecast, and comment the before-and-after figures. It does not need to block anything — seeing “this adds roughly 3.4 TB” is sufficient to prompt the conversation, and a conversation is all the control that is needed here.

Verification 

# 1. The forecast matches reality for the levels already seeded
python3 tile_forecast.py | head -4
aws s3 ls --summarize --recursive s3://portal-tiles/basemap/ | tail -1
#   expect: the forecast within a reasonable factor; tune AVG_TILE_BYTES from the real archive

# 2. Growth is tracked as a series, not read from a bill
aws cloudwatch get-metric-statistics --namespace AWS/S3 --metric-name BucketSizeBytes \
  --dimensions Name=BucketName,Value=portal-tiles Name=StorageType,Value=StandardStorage \
  --start-time "$(date -d '90 days ago' -Iseconds)" --end-time "$(date -Iseconds)" \
  --period 86400 --statistics Average | jq -r '.Datapoints | sort_by(.Timestamp) | .[-1].Average'

# 3. Lifecycle rules are actually applying
aws s3api list-objects-v2 --bucket portal-tiles --prefix basemap/16/ --max-items 5 \
  --query 'Contents[].StorageClass'
#   expect: STANDARD_IA for objects older than the transition

# 4. Expired deep tiles are re-rendered rather than 404ing
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
  "https://tiles.example.gov/wmts/basemap/17/65500/43600.png"
#   expect: 200, slower than a hit, and a second request served fast

Troubleshooting matrix

Symptom Likely cause Fix
Storage doubled after a cartography change A second style created a second full archive Forecast per style before publishing; consider vector tiles for restyling
The forecast is far off the measured size AVG_TILE_BYTES taken from a sparse area Measure the average from the densest part of the coverage
Deep-zoom tiles expire and requests start failing The renderer was retired or cannot serve that layer Confirm on-demand rendering works before adding an expiry rule
Growth is noticed only on the bill No size metric tracked over time Record bucket size daily as a series with the same retention as traffic
The archive is large and the hit ratio is poor Seeded uniformly rather than by request pattern Seed from the observed request distribution, not from the coverage polygon
Lifecycle transitions do not reduce cost Objects are small enough that per-object overhead dominates Check the minimum billable size for the target class before transitioning
A re-seed after a style change costs a full archive again The old archive was not expired alongside Expire superseded style prefixes as part of the publish

FAQ

How accurate does a forecast need to be?

Within a factor of two is enough to change a decision, which is all it is for. The purpose is to make “add one more zoom level” visibly a doubling rather than an increment, and that conclusion is robust to a great deal of imprecision in the average tile size.

Is it cheaper to store deep tiles or to render them on demand?

Almost always to render, for the deepest level or two, because so little of it is ever requested. The exception is a portal where deep zoom is the primary use — a utilities or cadastral viewer whose readers spend their whole session below zoom 17 — in which case the request distribution is different and should be measured rather than assumed.

Do vector tiles solve this?

They change its shape. A vector archive is style-independent, so a restyling costs nothing in storage instead of duplicating the archive — a substantial saving for a portal with several cartographic variants. The per-tile size becomes variable with feature density, which makes the forecast less arithmetic and more measurement.

Should the forecast include the fallback and replicated copies?

Yes, explicitly, because they are real stored bytes and they are easy to forget. A shallow fallback archive and a cross-region replica of it are small next to the main archive, and they belong in the same table so that nobody is surprised by a line item they did not know existed.

How often should this be revisited?

Quarterly, and before any change to coverage, zoom range or style count. The quarterly review catches drift; the pre-change forecast is what prevents the decision that produces the drift in the first place.

What is a reasonable average tile size to start from?

Measure it rather than adopt a figure: for a typical raster basemap style it commonly lands between eight and twenty kilobytes, and the spread across styles is wide enough that borrowing another portal’s number produces a forecast off by a factor of two. Compute the mean from a sample of the existing archive, and recompute it whenever the style changes.

Does compression at rest change the picture?

Somewhat, and less than expected for raster tiles, which are already compressed image formats and gain little from a second pass. Vector tiles compress well and are usually stored compressed already. The lever with real effect on raster archives is the format and quality setting itself, which is a cartographic decision with a visible outcome — and therefore one to make deliberately rather than as a storage optimisation.

Up one level: Autoscaling & Capacity Planning for Geospatial Workloads.