Geospatial Portal

Implementing OAI-PMH Resumption Tokens

A concrete procedure for driving OAI-PMH ListRecords through resumption-token pagination in Python, persisting a durable cursor so a crashed harvest resumes mid-stream, and proving the run harvested every record the server promised.

This guide is the hands-on companion to Incremental Metadata Harvesting and Change Detection and sits inside the wider Metadata Catalog Automation & Ingestion Workflows practice; read the parent guide first for why the harvest window is bounded by a saved watermark and how change detection classifies each record. Here the scope is narrow: the exact token loop, where the durable cursor is written, how to survive token expiry, and how to verify completeness against the server’s own count.

Prerequisites

  • Python 3.11+ with requests (or httpx) and lxml >= 5.0 for hardened XML parsing.
  • A reachable repository base URL with a successful Identify response confirming granularity and earliestDatestamp.
  • A supported metadataPrefix (from ListMetadataFormats) — commonly oai_dc or iso19139.
  • Durable cursor storage: a SQLite file, Redis key, or a PostGIS-backed table. In-memory state is not acceptable — a crash between pages must be recoverable.
  • Environment: OAI_BASE_URL, OAI_PREFIX, and HARVEST_STATE_DB set for the worker; write access to the cursor store.
  • Familiarity with the selective-harvest window (from/until/set) from the parent guide, since the token loop resumes within one such window.

The sequence below traces the token loop: issue the first windowed ListRecords, persist the returned token before processing the page, upsert each record, then re-request by token until the response carries none — restarting the window on badResumptionToken.

Resumption-token pagination loop A sequence diagram with three lifelines: Harvester, Cursor store, and OAI-PMH repository. The harvester issues ListRecords with metadataPrefix and from against the repository, which returns a page of records and a resumptionToken. The harvester persists that token to the cursor store, upserts the page of records, then re-issues ListRecords carrying only the resumptionToken. The loop repeats until the repository returns a page with no token, at which point the harvester clears the token and advances the watermark. A note states that a badResumptionToken response causes the harvester to discard the token and restart the windowed ListRecords from the saved from-cursor. Harvester Cursor store OAI-PMH repo ListRecords · prefix · from records + resumptionToken persist token (before processing) upsert page ListRecords · resumptionToken only loop while token present final page · no token clear token · advance watermark On badResumptionToken: discard token, restart ListRecords from the saved from-cursor

Step-by-step implementation

1. Issue the first windowed ListRecords

The opening request of a window carries metadataPrefix and from (plus optional until/set); it must never carry a token. Isolate one hardened, XXE-safe parser and reuse it for every page so a hostile or malformed payload cannot exhaust memory or reach the network.

import requests
import lxml.etree as etree

OAI_NS = {"oai": "http://www.openarchives.org/OAI/2.0/"}

# One reusable, XXE-safe parser for every page in the loop.
PARSER = etree.XMLParser(
    recover=True, no_network=True, resolve_entities=False, huge_tree=False,
)

def fetch_page(session, base_url, prefix, from_ts, token=None):
    """Fetch one ListRecords page. Token, when present, is the ONLY argument."""
    if token:
        params = {"verb": "ListRecords", "resumptionToken": token}
    else:
        params = {"verb": "ListRecords", "metadataPrefix": prefix, "from": from_ts}
    resp = session.get(base_url, params=params, timeout=30)
    resp.raise_for_status()
    return etree.fromstring(resp.content, PARSER)

2. Read the resumptionToken from the response

The token lives in //oai:resumptionToken. An element that is present but empty means “this was the last page” — an absent element and an empty element both terminate the loop. Read the completeListSize and cursor attributes here too; they drive the completeness check in verification.

def read_token(root):
    """Return (token_or_None, complete_list_size_or_None) from a ListRecords page."""
    node = root.find(".//oai:resumptionToken", OAI_NS)
    if node is None or not (node.text and node.text.strip()):
        return None, None                       # empty or absent -> window complete
    size = node.get("completeListSize")
    return node.text.strip(), (int(size) if size else None)

def read_error(root):
    """Return the OAI error code on an error response, else None."""
    err = root.find("oai:error", OAI_NS)
    return err.get("code") if err is not None else None

3. Loop issuing ListRecords by token until it is absent

The loop persists the token before processing each page, so a crash resumes from the page it was about to work, not the page it just finished. This ordering is the difference between a resumable harvest and one that silently skips a page on restart.

def harvest_window(session, base_url, prefix, state, upsert):
    """Page a full window with durable resumption. `state` is the cursor store."""
    saved = state.read()
    from_ts, token = saved["last_ts"], saved.get("token")
    seen = 0
    while True:
        root = fetch_page(session, base_url, prefix, from_ts, token)
        code = read_error(root)
        if code == "noRecordsMatch":
            break                               # nothing new in this window
        if code == "badResumptionToken":
            token = None                        # step 4: restart from the window
            state.commit(from_ts, token=None)
            continue
        token, complete_size = read_token(root)
        state.commit(from_ts, token=token)      # persist BEFORE processing the page
        for rec in root.iterfind(".//oai:record", OAI_NS):
            upsert(rec)                          # idempotent, keyed on oai:identifier
            seen += 1
        if token is None:
            break                                # window fully paged
    state.commit(new_watermark(), token=None)    # advance only after the window closes
    return seen, complete_size

4. Handle token expiration

Tokens carry a server-side TTL. A worker that stalls past it gets badResumptionToken on the next request. The only safe recovery — shown in the loop above — is to discard the token and reissue the windowed ListRecords from the saved from cursor. Because every record write is idempotent, replaying the window’s earlier pages is a no-op, not duplication. Never fabricate or mutate a token, and never resume from a partial per-record cursor: both desynchronize the local catalog against the source.

5. Idempotent upsert per record identifier

Redelivery is guaranteed by design — window overlap, token restarts, and expiry replays all re-present records — so the write must converge. Key every upsert on the stable oai:identifier and treat a status="deleted" header as a withdrawal, not an insert.

def upsert(record_el):
    """Upsert one OAI record keyed on its identifier; honor deletion tombstones."""
    header = record_el.find("oai:header", OAI_NS)
    identifier = header.findtext("oai:identifier", namespaces=OAI_NS)
    if header.get("status") == "deleted":
        withdraw_from_index(identifier)          # tombstone -> remove, do not insert
        return
    body = record_el.find("oai:metadata", OAI_NS)
    store_record(identifier, etree.tostring(body))   # ON CONFLICT(identifier) DO UPDATE

The cursor store backing state.commit should be a durable table or key. A minimal, correct shape keyed per endpoint:

{
  "endpoint": "https://catalog.agency.gov/oai",
  "metadata_prefix": "iso19139",
  "last_ts": "2026-07-01T00:00:00Z",
  "token": null,
  "errors": 0
}

Verification

Confirm the loop terminated cleanly, harvested every promised record, and is a no-op on re-run. Execute against a staging endpoint first.

# 1. The server's promised total for the window (completeListSize on the first token)
curl -s "https://catalog.agency.gov/oai?verb=ListRecords&metadataPrefix=iso19139&from=2026-07-01" \
  | xmllint --xpath 'string(//*[local-name()="resumptionToken"]/@completeListSize)' -
# -> e.g. 4820

# 2. The count actually stored after the harvest run
psql -At -c "SELECT count(*) FROM catalog_record WHERE harvested_at >= '2026-07-01';"
# -> 4820  (must equal completeListSize; a shortfall means a dropped page)

# 3. The cursor closed the window: token is NULL, watermark advanced
sqlite3 "$HARVEST_STATE_DB" "SELECT last_ts, token, errors FROM state;"
# -> 2026-07-13T00:00:00Z | (null) | 0

# 4. Re-running the same window is idempotent: row count does not change
python -m harvest.run --from 2026-07-01 --until 2026-07-02
psql -At -c "SELECT count(*) FROM catalog_record WHERE harvested_at >= '2026-07-01';"
# -> 4820  (unchanged: upserts collapsed the redelivered records)

A stored count equal to completeListSize, a null token with an advanced last_ts, and an unchanged row count on re-run together prove the paging loop is complete, crash-safe, and idempotent. Wire checks 1 and 2 into a pipeline gate so no promotion happens on a short harvest.

Troubleshooting matrix

Symptom Likely cause Fix
Loop never terminates resumptionToken element present-but-empty not treated as end-of-list Return None when the token text is empty or whitespace; break the loop
badArgument on the second page Token sent alongside from/metadataPrefix On any token request, send resumptionToken as the sole argument
badResumptionToken mid-window Token TTL exceeded while the worker stalled Discard the token, reissue windowed ListRecords from the saved from cursor
Restart skips a page Token persisted after processing instead of before Commit the token before upserting the page it belongs to
Stored count below completeListSize A page silently dropped, or until truncated the window Re-run the window from from; verify no exception swallowed a page
Duplicate rows after a crash Insert instead of upsert on the identifier Upsert keyed on oai:identifier; make replay a no-op
Deleted records reappear on re-harvest status="deleted" header inserted rather than withdrawn Branch on the header status; withdraw the identifier from the index

For the authoritative token and error-response semantics, the OAI-PMH 2.0 specification is definitive, and the lxml parsing documentation covers the production-safe parser tuning used above.

Up one level: Incremental Metadata Harvesting and Change Detection.