Geospatial Portal

Incremental Metadata Harvesting and Change Detection

Full re-harvests of a large federated catalog get slower every month and pound upstream endpoints with requests for records that have not changed since the last run. The teams that feel this are the platform engineers watching a nightly sync blow past its window, and the agency operators whose repository gets rate-limited by a partner portal re-reading its entire holdings. Incremental — or delta — harvesting is the discipline that reads only what changed, converges the local catalog to the source, and does so without ever replaying the whole corpus.

This guide sits within the Metadata Catalog Automation & Ingestion Workflows practice and assumes the harvest already runs on a schedule; the focus here is narrowing each cycle to the changed set, detecting that set reliably, and persisting enough state that a crash or an outage never forces a full rebuild.

The pipeline below reads a saved watermark, requests only the changed window, pages the response with resumption tokens, classifies each record as new, updated, or deleted, and applies it idempotently before advancing the cursor.

Incremental harvest and change-detection pipeline Flow runs left to right. A cursor store holds the last successful datestamp and in-flight token. It supplies a from/until watermark to a selective ListIdentifiers call scoped by set. That call feeds a resumption-token paging loop that self-repeats while a token is present. Each returned header passes into a change detector that classifies the record as new, updated (by checksum or sequence number), or deleted (tombstone). Classified records flow into an idempotent upsert keyed on the OAI identifier and then into the catalog index. Only after the whole batch commits does the pipeline write the advanced watermark back to the cursor store, shown as a dashed return path. Cursor store watermark · token ListIdentifiers from · until · set metadataPrefix Token paging loop while token present Change detector new · updated checksum · seq deleted · tombstone Idempotent upsert on id Catalog index Advance watermark after batch commit last_ts moves forward only when records are durable next page persist advanced cursor

Where Delta Harvesting Sits in the Ingestion Path

Incremental harvesting is the read side of the catalog automation stack: it decides which records leave the upstream repository, before any transform or validation gate ever sees them. Everything downstream — mapping, semantic validation, indexing — scales with the volume this stage admits, so a harvester that fails to narrow its window pushes the entire pipeline into a full-corpus reprocess on every cycle. The correct placement is a scheduled worker that owns a durable cursor per endpoint, pulls only the changed window, and hands normalized deltas to the transform stage described in CSW catalog schema mapping and validation.

The protocol most government and scientific portals expose for this is OAI-PMH. Its selective-harvest arguments — from, until, set, and metadataPrefix — are the primitives that turn a full sweep into a delta. A request scoped to from=<last successful datestamp> returns only records whose server-side datestamp advanced since the previous run, and a set restricts that further to a thematic or agency subtree. The pull mechanics, session hardening, and quarantine behavior that make this reliable in production are covered in automated metadata ingestion via OAI-PMH; this guide layers the change-detection logic on top of that transport.

The load argument is decisive. A federated catalog aggregating dozens of repositories cannot re-read millions of records nightly without either exhausting its own workers or tripping every upstream fair-use ceiling. Delta harvesting bounds each cycle to the change rate of the source rather than its total size, which is what keeps the sync window flat as the catalog grows.

Selective Harvesting with Datestamp Windows and Sets

The core delta request pins the metadata format and opens a datestamp window. Use ListIdentifiers rather than ListRecords for detection: it returns lightweight headers (identifier, datestamp, set membership, and deletion status) at a fraction of the payload, letting you decide what to fetch in full before spending bandwidth on it.

# harvest/select.py — build the selective ListIdentifiers request for one delta window.
# granularity must match the endpoint's Identify response: day vs full UTC timestamp.
def delta_params(prefix: str, from_ts: str, until_ts: str | None,
                 set_spec: str | None) -> dict:
    params = {"verb": "ListIdentifiers", "metadataPrefix": prefix, "from": from_ts}
    if until_ts:
        params["until"] = until_ts          # bound the window for catch-up shards
    if set_spec:
        params["set"] = set_spec            # restrict to an agency/thematic subtree
    return params

Two window rules prevent the classic delta bugs. First, normalize every timestamp to UTC and clamp from to the endpoint’s advertised earliestDatestamp, so a clock-skewed or over-eager cursor never asks for a window the server rejects with noRecordsMatch. Second, make the window slightly inclusive: overlap the new from with the previous run by one granularity unit and rely on idempotent upsert to absorb the duplicates. A strictly exclusive window risks dropping a record whose datestamp equals the boundary second.

Sets are the second axis of selectivity. Where a portal partitions holdings by set, running one cursor per (endpoint, set, metadataPrefix) tuple lets you harvest a high-churn subtree every 15 minutes while sweeping a static archive daily, instead of forcing a single interval on everything.

Resumption-Token Pagination

A datestamp window can still match more records than the server will return in one response. OAI-PMH solves this with resumption tokens: the response carries a resumptionToken element, and the client re-requests the same verb with only that token as its argument until the token is absent. The token is stateful and server-bound — you must not mix it with from, until, or metadataPrefix on the follow-up request.

The token also carries the completeness contract. Many repositories populate a completeListSize attribute and a cursor offset on the token, which together let the harvester verify it received every page rather than silently stopping on a truncated response. The full loop — durable cursor persistence, token-expiry recovery, and the completeness check — is the subject of the companion page on implementing OAI-PMH resumption tokens, which turns the paragraphs here into a crash-safe paging loop.

The operational hazard is token expiry. Tokens have a server-side TTL; if a worker stalls between pages past that TTL, the next request returns badResumptionToken. The only correct recovery is to discard the token and reissue the original windowed ListIdentifiers from the saved from cursor — never to guess a token or resume from a partial cursor, both of which desynchronize the local catalog.

Change-Detection Strategies

Selective harvesting tells you a record’s datestamp moved; it does not tell you what changed or whether the change is meaningful. Three strategies, layered, give a trustworthy signal.

Datestamp windows are the primary and cheapest signal. The server’s datestamp on each header advances when the record is created, modified, or deleted, so a from-bounded list is already the candidate change set. Its weakness is granularity: a day-granularity endpoint cannot distinguish two edits on the same date, which is why the window must be treated as candidates and confirmed downstream.

Content checksums or ETags confirm a real change. Compute a stable hash (SHA-256) over the normalized record body and store it beside the identifier. When a candidate arrives, re-fetch and re-hash: an unchanged hash means the datestamp moved for a non-substantive reason (a re-publish, a bulk touch) and the record can be skipped, sparing the transform and index stages. Where the endpoint exposes HTTP ETag/Last-Modified on GetRecord, a conditional request short-circuits the fetch entirely.

Sequence numbers are the strongest ordering signal when a source provides them. A monotonic per-record version or change-sequence counter lets the harvester apply a strict happens-before ordering, so an out-of-order or replayed delivery can be recognized and discarded rather than clobbering a newer state. Store the last-applied sequence per identifier and reject any incoming record whose sequence is not greater.

# harvest/detect.py — classify a candidate header against stored state.
# Returns the action the upsert stage should take for this identifier.
import hashlib

def classify(header, body: bytes, stored: dict | None) -> str:
    if header.deleted:
        return "delete"                       # tombstone -> withdraw from index
    if stored is None:
        return "insert"                       # unseen identifier
    digest = hashlib.sha256(body).hexdigest()
    if digest == stored["checksum"]:
        return "skip"                         # datestamp moved but content is identical
    if header.sequence is not None and header.sequence <= stored["sequence"]:
        return "skip"                         # stale/replayed delivery, keep newer state
    return "update"

DeletedRecord Semantics

A delta harvester that only inserts and updates will accumulate ghost records — entries deleted upstream that linger in the local index forever. OAI-PMH handles deletions through the deletedRecord policy each repository declares in its Identify response, and the correct client behavior depends entirely on which of three values it returns.

deletedRecord policy What the repository guarantees Required harvester behavior
persistent Deletions are kept permanently and always surface as status="deleted" headers in datestamp windows Trust incremental tombstones; withdraw the identifier on every deleted header and never full-scan for deletions
transient Deletions may appear as tombstones but are not guaranteed to persist across the retention window Trust tombstones when present, but reconcile with a periodic full identifier list to catch dropped deletions
no The repository does not report deletions at all Detect deletions yourself by diffing a periodic full ListIdentifiers snapshot against the local set and withdrawing the difference

The practical rule: read the deletedRecord value once from Identify, store it with the endpoint config, and let it select the deletion strategy. Treating a no-policy endpoint as if it were persistent is the most common cause of a catalog that slowly fills with records the source removed years ago.

Idempotent Upsert, Dedup, and Watermark Persistence

Delta application must converge to the same state no matter how many times a record is delivered, because overlapping windows, token restarts, and replays all cause redelivery by design. The two invariants are: every write is an upsert keyed on the OAI identifier, and the harvest watermark advances only after the batch is durably committed.

-- Idempotent upsert keyed on the stable OAI identifier.
-- Re-running the same delta batch is a no-op: same id, same checksum -> no change.
INSERT INTO catalog_record (oai_identifier, checksum, sequence, body, harvested_at)
VALUES (:id, :checksum, :sequence, :body, now())
ON CONFLICT (oai_identifier) DO UPDATE
   SET checksum    = EXCLUDED.checksum,
       sequence    = EXCLUDED.sequence,
       body        = EXCLUDED.body,
       harvested_at = EXCLUDED.harvested_at
 WHERE EXCLUDED.sequence > catalog_record.sequence   -- reject stale/out-of-order deltas
    OR catalog_record.sequence IS NULL;

Dedup is a consequence of that key, not a separate stage: two harvesters racing the same endpoint, or one worker replaying a window after a crash, both resolve to a single row. The WHERE guard on the sequence makes the upsert monotonic so a replayed older delivery cannot overwrite newer state.

Watermark persistence is what makes the whole cycle crash-safe. The cursor — last successful datestamp plus in-flight token — is written to durable storage, and the ordering is strict: commit the records first, then advance last_ts. Reversing that order means a crash between the two writes silently skips a window forever. When throughput demands more workers than one endpoint can feed, the fan-out and back-pressure model for that lives in scaling Celery pipelines for bulk ingestion, which keeps per-cursor ordering intact while parallelizing across endpoints.

Scheduling and Catch-Up After Downtime

Run the harvester as a long-lived scheduled worker, not a cron one-shot, so backoff timers, circuit-breaker state, and per-endpoint error counters survive between cycles. Each tick reads the cursor, harvests the window [last_ts, now], and — critically — the window is defined by the saved watermark, not by “the last hour.” That single property is what makes downtime recovery automatic: after an outage, the next run’s window simply spans the whole gap.

For a long gap, harvesting one enormous window can overwhelm both the source and the local transform stage. Shard the catch-up into bounded sub-windows with explicit until bounds and process them in order, advancing the watermark after each shard so partial progress survives a second failure. Because every write is idempotent, an over-generous catch-up window costs redundant reads, never corruption.

Two guardrails keep scheduled harvesting honest under sustained pressure. First, respect the upstream fair-use ceiling and honor Retry-After on 429/503, quarantining an endpoint after N consecutive failures rather than hammering it. Second, expose the cursor lag — wall-clock now minus last_ts — as a metric, so a stuck cursor becomes an alert instead of a silent gap discovered weeks later during an audit.

Operational Troubleshooting

Diagnose delta-harvest faults by symptom, the state that reveals the cause, and the config surface that fixes it:

Symptom Likely cause Where to look Fix
Every run re-harvests the full catalog from not applied, or watermark written before records committed Cursor last_ts vs. request params in the harvest log Send from=last_ts; commit records first, then advance the watermark
badResumptionToken mid-cycle Token TTL exceeded while the worker stalled between pages Token age vs. endpoint TTL Discard the token, reissue windowed ListIdentifiers from the saved from cursor
Deleted records linger in the index deletedRecord policy mishandled (treating no as persistent) Identify policy value vs. deletion strategy in config Diff a periodic full identifier snapshot for no/transient endpoints
Duplicate catalog rows after a partition Non-idempotent insert instead of upsert on the identifier Rows sharing an oai_identifier Upsert keyed on identifier; add the monotonic sequence guard
Records re-transform though nothing changed Datestamp moved on a non-substantive re-publish Stored checksum vs. re-fetched body hash Skip on checksum match before handing to the transform stage
noRecordsMatch when changes are expected from window ahead of server clock or past earliestDatestamp Window bounds vs. Identify earliestDatestamp Normalize to UTC; clamp to earliestDatestamp; allow a skew margin
Cursor stuck, lag climbing silently Endpoint quarantined but no alert wired Cursor lag metric vs. alert rules Alert on now - last_ts crossing an SLA threshold

Treat the cursor lag metric and an empty quarantine as the two health signals that prove the harvest is both current and clean; wire both into the same observability stack that watches the rest of the ingestion path. For the authoritative verb and error semantics, the OAI-PMH 2.0 specification governs selective harvesting and deletedRecord policy, and the OGC catalogue standards define the CSW discovery contract these harvested records ultimately serve.

Up one level: Metadata Catalog Automation & Ingestion Workflows.