Geospatial Portal

Tuning Elasticsearch Shards for Spatial Catalogs

This guide sizes the shards of a spatial catalogue index from its actual record count and query pattern, and explains why the usual defaults are wrong for this workload in both directions. It belongs to Search Indexing Optimization with Elasticsearch, within the Metadata Catalog Automation & Ingestion Workflows framework.

Prerequisites

  • An existing catalogue index with a representative record count and document size.
  • The real query set — text searches, spatial filters, facet aggregations — taken from logs.
  • An alias in front of the index, so a reshard is a swap rather than an outage.
  • Cluster metrics available, so the effect of a change is measurable rather than felt.

A catalogue is a small index with expensive queries

Most shard advice is written for logging workloads: enormous document counts, simple queries, time-based indices. A spatial catalogue is the opposite shape — tens or hundreds of thousands of documents, which is small, with queries that combine full-text scoring, spatial filtering and several facet aggregations, which is expensive per query.

Why logging shard advice does not transfer Logging and catalogue workloads compared by document count, query complexity, partitioning and the shard count each implies. PROPERTY LOGGING SPATIAL CATALOGUE documents billions, growing constantly tens to hundreds of thousands query shape filter and count, mostly text scoring + spatial filter + facets partitioning by time, naturally none — every query spans everything shards implied many, per index one, or very few

The bottom row is the practical conclusion and it surprises people: for a catalogue of a few hundred thousand records, a single primary shard is usually the right answer. Every additional shard adds a fan-out, a partial result set to merge and — for scored text queries — a distribution of term statistics across shards that makes relevance slightly worse.

Step-by-step implementation

1. Measure what you actually have 

# Document count and the real on-disk size per shard.
curl -s "http://es:9200/_cat/shards/catalogue?v&h=index,shard,prirep,docs,store"

# And the query mix, from the slow log or from the application's own metrics.
curl -s "http://es:9200/catalogue/_stats/search?filter_path=**.query_total,**.query_time_in_millis"

2. Choose the primary count from size, not from node count

Create the index with PUT /catalogue-v3 and this body:

{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 1,

      "refresh_interval": "30s",

      "sort.field": ["published"],
      "sort.order": ["desc"]
    }
  }
}

A refresh interval of thirty seconds rather than the default second is worth stating explicitly. A catalogue is not a real-time system: records arrive from a harvest, and a delay of half a minute before a new record is searchable is imperceptible to users while removing a substantial and continuous indexing cost.

3. Add shards only when a measured limit is reached

What justifies another primary shard Three valid reasons and three invalid ones for increasing the primary shard count. justifies more shards a shard approaching tens of gigabytes merges falling behind sustained indexing a document count in the hundreds of millions all three are size problems, measurable in advance does not a node was added to the cluster a query is slow because of an aggregation a general wish for parallelism each of these makes catalogue queries slower, not faster

4. Reshard behind the alias, never in place 

# Build the new index, populate it from the source of record, verify, then swap.
curl -XPUT "http://es:9200/catalogue-v4" -H 'Content-Type: application/json' \
  -d '{"settings":{"index":{"number_of_shards":2,"number_of_replicas":1}}}'

python3 manage.py rebuild_search_index --index catalogue-v4

# Run the fixed query set against the new index BY NAME before any swap.
python3 tools/query_set.py --index catalogue-v4 --expect tools/query_expectations.json

# Atomic alias swap — the application never learns that anything changed.
curl -XPOST "http://es:9200/_aliases" -H 'Content-Type: application/json' -d '{
  "actions": [
    {"remove": {"index": "catalogue-v3", "alias": "catalogue"}},
    {"add":    {"index": "catalogue-v4", "alias": "catalogue"}}
  ]}'

Replicas serve queries; primaries serve writes

The other half of the sizing decision is replicas, and it is the one that actually helps a read-heavy catalogue. Each replica is a full copy that can serve searches, so replica count is the lever for query throughput, while primary count is the lever for index size and indexing throughput.

Primaries and replicas do different jobs Primary shards for size and indexing, replicas for query throughput and resilience, with the cost and changeability of each. primary shards bound index size and indexing rate fixed at creation changing them means reindexing more of them makes scored queries slightly worse replicas bound query throughput and resilience changeable at any time each costs a full copy of the storage the lever that actually helps a read-heavy catalogue

One further habit is worth building in before any of this is tuned: record the shard configuration, document count, index size and the query set’s p50 and p95 together, as one row, every time the index is rebuilt. A catalogue’s shape changes slowly, and the question that arrives eighteen months later — whether the current configuration was ever measured or merely inherited — is answerable only if somebody kept the row. It costs a line in the rebuild script and it is the difference between tuning and guessing.

Verification 

# 1. The index has the intended shape
curl -s "http://es:9200/catalogue/_settings?filter_path=**.number_of_shards,**.number_of_replicas" | jq .

# 2. Query latency improved, or at least did not regress
python3 tools/query_set.py --index catalogue --report p50,p95
#   compare against the recorded figures from before the change

# 3. Relevance did not change for the fixed query set
python3 tools/query_set.py --index catalogue --expect tools/query_expectations.json
#   expect: the same top result for every query in the set

# 4. Shard sizes are within the intended band
curl -s "http://es:9200/_cat/shards/catalogue?v&h=shard,prirep,docs,store"

# 5. The alias points where you think it does
curl -s "http://es:9200/_cat/aliases/catalogue?v"

Check 3 is the one to insist on. A reshard changes how term statistics are distributed, and relevance can shift subtly — the same query returning a different top result is a change users notice and nobody attributes to a shard setting.

Troubleshooting matrix

Symptom Likely cause Fix
Queries got slower after adding shards Fan-out and merge cost exceeding any parallelism gain Reduce primaries; a small catalogue usually wants one
Relevance changed after a reshard Term statistics distributed differently across shards Verify with the fixed query set before the swap; consider a single primary
Indexing is slow during a harvest Refresh interval at the default of one second Raise it for the catalogue; a delay of seconds is imperceptible
The cluster has thousands of tiny shards An index per source, or per month, on a small catalogue Consolidate into one index; use a field to distinguish sources
A reshard caused a search outage Reindexing in place rather than behind an alias Always build a new index and swap the alias
Adding a replica did not improve throughput Only one node, so the replica is unassigned Check shard allocation; a replica needs somewhere else to live
Heap pressure during faceted searches Aggregations over high-cardinality fields, unrelated to shards Limit facet cardinality; shards are not the lever here

FAQ

Is one primary shard really enough?

For a catalogue of a few hundred thousand records, yes, and it is usually faster than the alternative. The rule of thumb worth remembering is that a shard should hold something in the low tens of gigabytes before a second one is justified — most catalogues never approach that, because metadata records are small.

Should there be an index per source?

No. It multiplies shards, complicates cross-source search, and buys nothing that a source field and a filter do not provide. The pattern is tempting because it makes a source’s records easy to delete; a delete-by-query is a better answer than a structural decision that affects every query.

How many replicas?

At least one for resilience, and beyond that from measured query load: each additional replica adds query capacity and a full copy of the storage. Because it can be changed at any time without reindexing, this is the setting to adjust experimentally — unlike primaries, which cannot be changed in place.

What about time-based indices?

They suit data with a natural time partition and queries that usually target recent data. A catalogue has neither: a search for parcels is as likely to match a record from four years ago as one from last week, so every query would fan out to every index and pay the cost for nothing.

Does the spatial filter influence shard sizing?

Not directly — a geo filter is evaluated within each shard and does not change how many are appropriate. What it does affect is query cost per shard, which is a reason to keep the shard count low: each additional shard repeats the filter setup and returns a partial result to merge.

How should a reshard be scheduled?

Outside a harvest window, behind an alias, with the query set run against the new index by name before the swap. And decide in advance what happens to records indexed during the rebuild — either dual-index for the duration or pause ingestion, as with any alias-based reindex.

It is usually the same index, and it should be. A separate index for the map interface means two copies to keep synchronised and two places for relevance to diverge, for no benefit — the spatial filter and the text query run against the same documents either way. Where the two genuinely need different analysis, use different fields in one index rather than different indices.

Does the sort field in the index settings help?

It helps queries that sort in the same order, by allowing early termination, and it costs a little at index time. For a catalogue whose default listing is newest-first it is usually worth having; for one whose default is relevance-scored it does nothing, because the scoring order is not known in advance.

How does the shard count interact with a full reindex?

More primaries make a reindex faster, because the write is parallelised, and slower to query afterwards for a small catalogue. That trade is real and almost always resolves in favour of query speed: a reindex happens occasionally and a query happens thousands of times a day.

Up one level: Search Indexing Optimization with Elasticsearch.