Geospatial Portal

Serving OGC API - Records Alongside CSW

This guide publishes an OGC API - Records interface next to an existing CSW endpoint, served from the same record store, so modern clients get JSON and existing integrations keep working. It belongs to CSW Catalog Schema Mapping & Validation, within the Metadata Catalog Automation & Ingestion Workflows framework.

Prerequisites

  • An existing CSW endpoint with known consumers, and a month of access logs identifying them.
  • Records in a normalised internal form, not stored only as CSW response XML.
  • A search index able to serve both interfaces, per tuning Elasticsearch shards for spatial catalogs.
  • A decision, written down, about how long CSW will be supported.

Two interfaces, one record store

The mistake that makes this expensive is treating the new interface as a new catalogue. It is a second projection of the same records, and everything upstream of the projection — harvesting, crosswalks, validation, scoring, deduplication — is shared.

Two projections of one record store A shared ingestion pipeline and record store, with separate CSW and OGC API projections at the edge. harvest + crosswalk validation, scoring, dedup normalised records one store, one truth CSW projection XML, for existing clients OGC API projection JSON, for modern ones same records same records Storing CSW XML as the record of truth is what makes a second interface expensive — the JSON then has to be derived from a serialisation.

Step-by-step implementation

1. Publish the landing page and conformance declaration

A GET on the landing page returns:

{
  "title": "Agency Geospatial Catalogue",
  "description": "Metadata for datasets published by the agency and its partners.",
  "links": [
    {"rel": "self",        "type": "application/json", "href": "https://portal.example.gov/api/records/"},
    {"rel": "service-desc","type": "application/vnd.oai.openapi+json;version=3.0",
     "href": "https://portal.example.gov/api/records/api"},
    {"rel": "conformance", "type": "application/json", "href": "https://portal.example.gov/api/records/conformance"},
    {"rel": "data",        "type": "application/json", "href": "https://portal.example.gov/api/records/collections"}
  ]
}

And GET /api/records/conformance returns:

{
  "conformsTo": [
    "http://www.opengis.net/spec/ogcapi-common-1/1.0/conf/core",
    "http://www.opengis.net/spec/ogcapi-records-1/1.0/conf/core",
    "http://www.opengis.net/spec/ogcapi-records-1/1.0/conf/searchable-catalog",
    "http://www.opengis.net/spec/ogcapi-features-1/1.0/conf/oas30",
    "http://www.opengis.net/spec/ogcapi-features-1/1.0/conf/geojson"
  ]
}

Declare only what is actually implemented. A conformance class advertised and not honoured is worse than an absent one, because a client will use it and fail in a way that looks like a portal bug.

2. Project a record into the JSON form 

# projection.py — one function per interface, both from the same internal record.
def to_ogcapi_record(r: dict) -> dict:
    return {
        "id": r["identifier"],
        "type": "Feature",
        "geometry": bbox_to_geometry(r["extent"]),
        "time": temporal_to_interval(r.get("temporal_extent")),
        "properties": {
            "title": r["title"],
            "description": r.get("abstract"),
            "keywords": r.get("keywords", []),
            "created": r.get("created"),
            "updated": r.get("modified"),
            "type": "dataset",
            "license": r.get("licence"),
            # Provenance is exposed rather than hidden: users of an aggregating
            # catalogue need to know which organisation asserted this.
            "providers": [{"name": r["provenance"]["source_name"], "roles": ["publisher"]}],
        },
        "links": [
            {"rel": "self", "type": "application/geo+json",
             "href": f"https://portal.example.gov/api/records/collections/main/items/{r['identifier']}"},
            {"rel": "alternate", "type": "application/xml",
             "title": "CSW representation",
             "href": f"https://portal.example.gov/csw?service=CSW&version=2.0.2&request=GetRecordById&id={r['identifier']}"},
            *[{"rel": "enclosure", "href": d["url"], "type": d.get("format")}
              for d in r.get("distributions", [])],
        ],
    }

The alternate link between the two representations is worth including in both directions. It makes the relationship explicit to a client that finds one and needs the other, and it is the cheapest possible migration aid.

The same search, reached two ways CSW parameters and OGC API parameters mapped onto a common internal search request. CSW OGC API - RECORDS INTERNAL ogc:BBOX filter bbox=minx,miny,maxx,maxy spatial filter AnyText PropertyIsLike q= full-text query startPosition / maxRecords offset / limit pagination TempExtent_begin / _end datetime= temporal filter

Routing both interfaces through one internal search is what keeps them consistent. Two independently implemented query paths diverge within months, and the divergence surfaces as a user reporting that the same search gives different results depending on which endpoint their tool uses.

4. Keep the CSW endpoint honest while it lives 

# Both interfaces, same origin, with the older one clearly still supported.
location /csw {
    proxy_pass http://catalogue_backend;
    add_header Link '</api/records/>; rel="alternate"; type="application/json"; title="OGC API - Records"' always;
}

location /api/records/ {
    proxy_pass http://catalogue_backend;
    add_header Link '</csw>; rel="alternate"; type="application/xml"; title="CSW 2.0.2"' always;
    add_header Access-Control-Allow-Origin "*" always;   # a JSON API is read by browsers
}

Deciding when CSW can be retired

The answer is not a date chosen in advance; it is a measurement. CSW consumers in an agency estate are long-lived, often unattended, and frequently unknown until they stop working.

Retirement is a measurement, not a date Measure usage, contact consumers, then retire — with the failure mode of skipping the first step. 1 · measure a month of logs, grouped by user agent and address 2 · contact a migration path and a proposed timeline 3 · retire only when remaining traffic is identified and has agreed Skipping step one is the classic failure: the endpoint is retired on a schedule, and a statutory reporting integration nobody knew about stops silently — discovered weeks later, by the organisation that depended on it.

A note on where the effort actually goes. Teams expect the specification work to dominate and find that it does not: the routes, the landing page and the conformance declaration are a few days of work against a well-specified target. What takes longer is the discipline underneath — making the internal record the source of truth rather than the CSW serialisation, routing both interfaces through one search implementation, and measuring who uses the old endpoint before proposing to retire it. Those three are where the cost and the value both sit, and none of them is visible in the specification.

Verification 

# 1. The landing page, conformance and collections all resolve
for P in "" conformance collections; do
  curl -s -o /dev/null -w "%{http_code} /api/records/$P\n" "https://portal.example.gov/api/records/$P"
done
#   expect: 200 for each

# 2. A record is retrievable through BOTH interfaces, with matching content
ID=parcels-2026
curl -s "https://portal.example.gov/api/records/collections/main/items/$ID" | jq -r '.properties.title'
curl -s "https://portal.example.gov/csw?service=CSW&version=2.0.2&request=GetRecordById&id=$ID" \
  | xmllint --xpath 'string(//*[local-name()="title"])' -
#   expect: identical titles

# 3. The same search returns the same records through both
curl -s "https://portal.example.gov/api/records/collections/main/items?q=parcels&limit=5" | jq -r '.features[].id' | sort > a.txt
python3 tools/csw_search.py --any-text parcels --max 5 | sort > b.txt
diff a.txt b.txt
#   expect: no difference

# 4. Each response advertises the other representation
curl -sI "https://portal.example.gov/api/records/" | grep -i '^link:'
curl -sI "https://portal.example.gov/csw" | grep -i '^link:'

# 5. Conformance claims are honoured
curl -s "https://portal.example.gov/api/records/collections/main/items?bbox=-1.4,50.8,-1.2,51.0&limit=2" | jq '.numberReturned'
#   expect: a number, and only records intersecting that box

Troubleshooting matrix

Symptom Likely cause Fix
The two interfaces return different results for the same search Two independent query implementations Route both through one internal search request
JSON records lack fields present in the XML Projection written from the CSW output rather than the internal record Project both from the normalised record
Browser clients fail with a CORS error No allow-origin on the JSON endpoint Add it deliberately for the read-only API
Clients follow a conformance class that does not work Declared without being implemented Declare only what is implemented and tested
Pagination behaves differently between interfaces Offset semantics differing by one Test the boundary explicitly; CSW start positions are one-based
Retiring CSW broke an unknown integration Retired on a schedule rather than on measurement Measure consumers first; contact before removing
Extents differ between representations One derived from a union, the other from the primary extent Derive both from the same field in the record

FAQ

Does this mean maintaining two catalogues?

No — two projections and one catalogue. The ingestion, crosswalk, validation, scoring and deduplication work is shared, and the additional cost is a serialisation layer and a set of routes. That is only true if the internal record is the source of truth; if CSW XML is what is stored, the second interface is genuinely a second implementation.

Which interface should the portal’s own UI use?

The JSON one, because it is easier to consume from a browser and it exercises the newer path continuously. Using it internally also means the interface most likely to have a bug is the one being used constantly by the people who can fix it.

How long should both run in parallel?

Until the measurement says otherwise. In an agency estate, a year is not unusual, because the consumers are desktop GIS installations and unattended integrations on their own upgrade cycles. The cost of running both is low; the cost of retiring the old one early is a partner’s statutory process.

Should the identifiers be the same in both?

Yes, absolutely. A record must have one identifier however it is retrieved, or the two interfaces describe different worlds and no consumer can move between them. This is the single most important consistency requirement in the whole exercise.

What about the search behaviour differing subtly?

It will, unless both go through one internal search. Text scoring, stemming and stop-word handling are easy to configure twice and hard to keep identical, and the divergence appears as a user reporting that a dataset is findable in one tool and not another — which is a genuinely confusing bug to receive.

Does OGC API - Records replace the ISO record?

No. It is another representation, and a rich ISO record remains the fuller description — the JSON projection carries what the specification defines, which is deliberately smaller. Keep the original record available through the alternate link so nothing is lost for the consumers who need the detail.

How should the two interfaces be versioned as the record model evolves?

Version the internal record model, not the interfaces, and let each projection decide how to express a change. Adding a field is invisible to both; changing the meaning of one is a model version bump that both projections must be updated for together. Versioning the interfaces separately produces a situation where a record retrieved as JSON and as XML describes the same dataset differently, which is the one outcome this whole arrangement exists to avoid.

Should the JSON interface be rate-limited differently?

Yes, and usually more generously, because a browser-based client legitimately makes many small requests where a CSW client makes few large ones. Apply the same tenant-aware limiting described in the API gateway topic, with a budget sized for the interactive pattern rather than for the batch one.

Up one level: CSW Catalog Schema Mapping & Validation.