Geospatial Portal

Configuring Elasticsearch Analyzers for Place Names

A procedure for building a custom Elasticsearch analyzer that folds accents, normalizes punctuation, expands place-name variants, and supports prefix autocomplete for gazetteer fields.

This page is the task-level companion to Search Indexing Optimization with Elasticsearch and belongs to the wider Metadata Catalog Automation & Ingestion Workflows practice; read the parent guide first if you need the reasoning behind field-level analysis choices and the trade-offs between index-time and query-time work. The scope here is narrow: how one gazetteer field should be tokenized so a user typing “st marys” finds “Saint Mary’s”, “São Paulo” survives ASCII folding, and prefix matches return before the keystroke lands. Left with the default standard analyzer, place-name search silently drops diacritics inconsistently, treats “Mt.” and “Mount” as unrelated terms, and cannot autocomplete without a wildcard scan.

Prerequisites

Confirm the following before editing index settings. Each maps to a recurring cause of a place-name field that scores poorly or fails to autocomplete.

  • Elasticsearch 8.x (settings and mappings below use 8.x syntax; the analysis block is unchanged since 7.x but reindex mechanics differ).
  • Cluster or index privileges to create indices, put mappings, and run _reindex — analyzer changes on a live field require a rebuild, not an in-place update.
  • A synonym source for place-name variants and abbreviations (St → Saint, Mt → Mount, Ft → Fort, directional and administrative abbreviations), maintained as a file synonym set or an inline list under version control.
  • Validated ISO 19115 records already flowing from the ingestion gate described in Validating ISO 19115 Metadata Before Ingestion, so the geographicIdentifier / place-name text you index is already clean.
  • A staging index you can throw away — never prototype analyzer changes against the production alias.
  • Awareness that edge_ngram belongs at index time only: applying it as the search analyzer double-expands prefixes and destroys relevance.

The core idea is a split-analyzer field. At index time you generate accent-folded, synonym-expanded terms plus edge-ngram prefixes so the inverted index already holds every substring a user might type. At query time you run a lighter analyzer that folds accents and applies synonyms but does not re-emit ngrams, so a query matches the stored prefixes instead of exploding into more of them.

Place-name analyzer token pipeline A left-to-right pipeline for the input "Saint Mary's, Île". The raw string first meets a character filter that removes apostrophes and commas, then a whitespace-style tokenizer splits it into words, then a lowercase filter, an ASCII-folding filter that turns Île into ile, a synonym filter that maps Saint to St, and finally an edge n-gram filter that emits leading prefixes. The rightmost column lists the resulting indexed terms including sa, sai, saint, st, mar, mary, il and ile. raw Saint Mary's, Île char_filter strip . ' , → Saint Marys Île tokenizer [Saint] [Marys] [Île] token filters lowercase asciifolding → ile synonym Saint→st edge_ngram indexed terms sa · sai · sain · saint st · s ma · mar · mary · marys il · ile

Step-by-step implementation

1. Define the custom analyzer in index settings

Build one index-time analyzer, place_name_index, and a lighter query-time analyzer, place_name_search. Both share a punctuation-stripping char_filter and the lowercase, asciifolding, and synonym token filters; only the index analyzer appends edge_ngram. Preserving the original token (preserve_original on the folding filter) keeps an exact-diacritic term available for boosting.

{
  "settings": {
    "index": {
      "max_ngram_diff": 18,
      "analysis": {
        "char_filter": {
          "strip_place_punct": {
            "type": "pattern_replace",
            "pattern": "[.'`,]",
            "replacement": ""
          }
        },
        "filter": {
          "fold_keep_original": {
            "type": "asciifolding",
            "preserve_original": true
          },
          "place_synonyms": {
            "type": "synonym",
            "lenient": true,
            "synonyms": [
              "st, saint",
              "mt, mount",
              "ft, fort",
              "ste, sainte",
              "n, north",
              "s, south"
            ]
          },
          "place_edge_ngram": {
            "type": "edge_ngram",
            "min_gram": 2,
            "max_gram": 20
          }
        },
        "analyzer": {
          "place_name_index": {
            "type": "custom",
            "char_filter": ["strip_place_punct"],
            "tokenizer": "standard",
            "filter": ["lowercase", "fold_keep_original", "place_synonyms", "place_edge_ngram"]
          },
          "place_name_search": {
            "type": "custom",
            "char_filter": ["strip_place_punct"],
            "tokenizer": "standard",
            "filter": ["lowercase", "fold_keep_original", "place_synonyms"]
          }
        }
      }
    }
  }
}

2. Bind the analyzer to the mapping with distinct index and search analyzers

Point the gazetteer field at place_name_index for analyzer and place_name_search for search_analyzer. This is the pivot of the whole design: the field is populated with ngram prefixes at write time, but a query is analyzed without ngrams so it matches those stored prefixes instead of generating a second layer of them. Add a keyword sub-field for exact filtering, sorting, and aggregations.

{
  "mappings": {
    "properties": {
      "place_name": {
        "type": "text",
        "analyzer": "place_name_index",
        "search_analyzer": "place_name_search",
        "fields": {
          "raw": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

3. Create the staging index and reindex existing metadata

Analyzer and mapping changes cannot be applied to a populated field in place. Create the new index with the settings and mapping above, then copy documents with the Reindex API, which re-analyzes every source value through the new pipeline. Run it asynchronously for large catalogs and track the returned task.

# Create the new index carrying the analyzer + mapping (settings.json holds both blocks)
curl -sS -X PUT "http://localhost:9200/gazetteer_v2" \
  -H 'Content-Type: application/json' \
  --data-binary @settings-and-mappings.json

# Reindex from the live index; async so the client does not block on a large corpus
curl -sS -X POST "http://localhost:9200/_reindex?wait_for_completion=false" \
  -H 'Content-Type: application/json' -d '{
    "source": { "index": "gazetteer_v1" },
    "dest":   { "index": "gazetteer_v2" }
  }'
# -> {"task":"oTUltX4IQMOUUVeiohTt8A:12345"}

4. Move the alias and validate token output with the _analyze API

Once the reindex task reports "completed": true, atomically swing the read alias from the old index to the new one so search traffic never sees a half-built index. Then confirm the analyzer emits what you expect using _analyze, which runs text through the exact configured chain without indexing anything.

# Atomic alias swap: readers cut over from v1 to v2 in one cluster-state update
curl -sS -X POST "http://localhost:9200/_aliases" \
  -H 'Content-Type: application/json' -d '{
    "actions": [
      { "remove": { "index": "gazetteer_v1", "alias": "gazetteer" } },
      { "add":    { "index": "gazetteer_v2", "alias": "gazetteer" } }
    ]
  }'

# Inspect index-time tokens: expect folded, synonym-expanded, ngram prefixes
curl -sS -X GET "http://localhost:9200/gazetteer_v2/_analyze" \
  -H 'Content-Type: application/json' -d '{
    "analyzer": "place_name_index",
    "text": "Saint Mary's"
  }'

Test the analyser against names you actually hold

An analyser is a chain of transformations, and reasoning about a chain from its configuration is unreliable — the interactions between a normaliser, a token filter and a synonym list are exactly where the surprises live. The way to know what a chain does is to run place names through it and read the tokens that come out, and the names to run are the ones in the catalogue rather than the ones in the documentation.

Four categories of name every analyser test should include Four rows naming a category of place name, the property that makes it awkward, and the assertion to make on the produced tokens. CATEGORY WHY IT IS AWKWARD ASSERT THAT diacritics users type them, or do not both spellings produce the same token hyphens and apostrophes tokenisers disagree about them the whole name still matches as a phrase multi-word with articles stop-word filters remove the article the name survives with its order intact also a common word matches unrelated prose the place field outranks the abstract

Build the test set by sampling real names from the catalogue’s own place index rather than inventing examples, and keep it in version control next to the analyser definition. Every future change to the chain then has an immediate, readable answer to the question that otherwise gets settled by opinion: which of these names does this change break?

Verification

Confirm the analyzer, the split index/search behavior, and the alias in isolation. Run each against the staging index first.

# 1. Query-time analysis must NOT emit ngrams — expect tokens: saint, st, marys
curl -sS "http://localhost:9200/gazetteer_v2/_analyze" \
  -H 'Content-Type: application/json' -d '{"analyzer":"place_name_search","text":"St Marys"}' \
  | python -c "import sys,json; print([t['token'] for t in json.load(sys.stdin)['tokens']])"
# -> ['st', 'saint', 'marys']   (no 'ma','mar' prefixes)

# 2. ASCII folding keeps both folded and original: expect 'sao' and 'são'
curl -sS "http://localhost:9200/gazetteer_v2/_analyze" \
  -H 'Content-Type: application/json' -d '{"field":"place_name","text":"São"}'

# 3. A prefix query returns the full place name (autocomplete works)
curl -sS "http://localhost:9200/gazetteer/_search" \
  -H 'Content-Type: application/json' -d '{"query":{"match":{"place_name":"sao pau"}}}' \
  | python -c "import sys,json; print(json.load(sys.stdin)['hits']['total'])"
# -> {'value': 1, 'relation': 'eq'}

# 4. The alias points only at v2
curl -sS "http://localhost:9200/_alias/gazetteer?pretty"

A ngram-free token list from the search analyzer, both folded and original tokens for accented input, a non-zero hit on a partial prefix, and an alias resolving to gazetteer_v2 together confirm the field is analyzed correctly and cut over cleanly.

Index-time and search-time analysis are different decisions

The same analyser is usually applied when a document is indexed and when a query is parsed, and that default is right often enough that the distinction goes unnoticed — until a synonym change produces results nobody can explain. The two are separate settings and they behave differently: an index-time change requires reindexing to take effect, while a search-time change applies to the next query.

Applying synonyms at index time or at search time Two columns comparing where synonym expansion happens, when a change takes effect, and the trade-off each carries. At index time expansions are baked into stored tokens a change needs a full reindex documents indexed earlier keep old rules fast queries, slow to correct At search time the query expands instead of the document a change applies to the next query every document sees the same rules slightly heavier queries, easy to correct

For place-name synonyms — which are revised as gazetteers are corrected and as users report misses — search-time expansion is usually the better default, because it makes each correction a configuration change rather than a reindex. Keep index-time analysis for the transformations that are genuinely stable, such as case and diacritic normalisation, where the reindex cost is paid once and the rules do not move.

Troubleshooting matrix

Symptom Likely cause Fix
Every query matches far too many documents edge_ngram filter applied as the search_analyzer too Set search_analyzer to place_name_search, which omits edge_ngram; reindex
illegal_argument_exception: max_ngram_diff max_gram - min_gram exceeds the index default of 1 Raise index.max_ngram_diff to cover your gram range (e.g. 18)
Accented names never match ASCII input asciifolding missing or after a filter that already split tokens Place asciifolding in the chain right after lowercase on both analyzers
“St” and “Saint” treated as different terms Synonym filter absent from the search analyzer Include place_synonyms in place_name_search, not only the index analyzer
Mapping update rejected on a live field analyzer cannot change on a populated field in place Create a new index and _reindex; swing the alias when complete
Reindex leaves stale accented docs unfolded Source copied without re-analysis (e.g. op_type=index on same mapping) Reindex into the new index so values pass the new analyzer
Autocomplete misses the first one or two letters min_gram set to 2 while UI queries single characters Lower min_gram, or gate the UI to fire after two characters
Synonym file edit not reflected in results Analyzer settings are read at index-open time Close/open the index or reindex; reload search analyzers if using a synonym file set

For authoritative behavior, the Elasticsearch analysis reference documents each filter, and the OGC Catalogue Service (CSW) standard defines the geographic-identifier semantics these place-name fields carry.

Up one level: Search Indexing Optimization with Elasticsearch.