Geospatial Portal

Implementing OAI-PMH Resumption Tokens

A concrete procedure for driving OAI-PMH ListRecords through resumption-token pagination in Python, persisting a durable cursor so a crashed harvest resumes mid-stream, and proving the run harvested every record the server promised.

This guide is the hands-on companion to Incremental Metadata Harvesting and Change Detection and sits inside the wider Metadata Catalog Automation & Ingestion Workflows practice; read the parent guide first for why the harvest window is bounded by a saved watermark and how change detection classifies each record. Here the scope is narrow: the exact token loop, where the durable cursor is written, how to survive token expiry, and how to verify completeness against the server’s own count.

Prerequisites

  • Python 3.11+ with requests (or httpx) and lxml >= 5.0 for hardened XML parsing.
  • A reachable repository base URL with a successful Identify response confirming granularity and earliestDatestamp.
  • A supported metadataPrefix (from ListMetadataFormats) — commonly oai_dc or iso19139.
  • Durable cursor storage: a SQLite file, Redis key, or a PostGIS-backed table. In-memory state is not acceptable — a crash between pages must be recoverable.
  • Environment: OAI_BASE_URL, OAI_PREFIX, and HARVEST_STATE_DB set for the worker; write access to the cursor store.
  • Familiarity with the selective-harvest window (from/until/set) from the parent guide, since the token loop resumes within one such window.

The sequence below traces the token loop: issue the first windowed ListRecords, persist the returned token before processing the page, upsert each record, then re-request by token until the response carries none — restarting the window on badResumptionToken.

Resumption-token pagination loop A sequence diagram with three lifelines: Harvester, Cursor store, and OAI-PMH repository. The harvester issues ListRecords with metadataPrefix and from against the repository, which returns a page of records and a resumptionToken. The harvester persists that token to the cursor store, upserts the page of records, then re-issues ListRecords carrying only the resumptionToken. The loop repeats until the repository returns a page with no token, at which point the harvester clears the token and advances the watermark. A note states that a badResumptionToken response causes the harvester to discard the token and restart the windowed ListRecords from the saved from-cursor. Harvester Cursor store OAI-PMH repo ListRecords · prefix · from records + resumptionToken persist token (before processing) upsert page ListRecords · resumptionToken only loop while token present final page · no token clear token · advance watermark On badResumptionToken: discard token, restart ListRecords from the saved from-cursor

Step-by-step implementation

1. Issue the first windowed ListRecords

The opening request of a window carries metadataPrefix and from (plus optional until/set); it must never carry a token. Isolate one hardened, XXE-safe parser and reuse it for every page so a hostile or malformed payload cannot exhaust memory or reach the network.

import requests
import lxml.etree as etree

OAI_NS = {"oai": "http://www.openarchives.org/OAI/2.0/"}

# One reusable, XXE-safe parser for every page in the loop.
PARSER = etree.XMLParser(
    recover=True, no_network=True, resolve_entities=False, huge_tree=False,
)

def fetch_page(session, base_url, prefix, from_ts, token=None):
    """Fetch one ListRecords page. Token, when present, is the ONLY argument."""
    if token:
        params = {"verb": "ListRecords", "resumptionToken": token}
    else:
        params = {"verb": "ListRecords", "metadataPrefix": prefix, "from": from_ts}
    resp = session.get(base_url, params=params, timeout=30)
    resp.raise_for_status()
    return etree.fromstring(resp.content, PARSER)

2. Read the resumptionToken from the response

The token lives in //oai:resumptionToken. An element that is present but empty means “this was the last page” — an absent element and an empty element both terminate the loop. Read the completeListSize and cursor attributes here too; they drive the completeness check in verification.

def read_token(root):
    """Return (token_or_None, complete_list_size_or_None) from a ListRecords page."""
    node = root.find(".//oai:resumptionToken", OAI_NS)
    if node is None or not (node.text and node.text.strip()):
        return None, None                       # empty or absent -> window complete
    size = node.get("completeListSize")
    return node.text.strip(), (int(size) if size else None)

def read_error(root):
    """Return the OAI error code on an error response, else None."""
    err = root.find("oai:error", OAI_NS)
    return err.get("code") if err is not None else None

3. Loop issuing ListRecords by token until it is absent

The loop persists the token before processing each page, so a crash resumes from the page it was about to work, not the page it just finished. This ordering is the difference between a resumable harvest and one that silently skips a page on restart.

def harvest_window(session, base_url, prefix, state, upsert):
    """Page a full window with durable resumption. `state` is the cursor store."""
    saved = state.read()
    from_ts, token = saved["last_ts"], saved.get("token")
    seen = 0
    while True:
        root = fetch_page(session, base_url, prefix, from_ts, token)
        code = read_error(root)
        if code == "noRecordsMatch":
            break                               # nothing new in this window
        if code == "badResumptionToken":
            token = None                        # step 4: restart from the window
            state.commit(from_ts, token=None)
            continue
        token, complete_size = read_token(root)
        state.commit(from_ts, token=token)      # persist BEFORE processing the page
        for rec in root.iterfind(".//oai:record", OAI_NS):
            upsert(rec)                          # idempotent, keyed on oai:identifier
            seen += 1
        if token is None:
            break                                # window fully paged
    state.commit(new_watermark(), token=None)    # advance only after the window closes
    return seen, complete_size

4. Handle token expiration

Tokens carry a server-side TTL. A worker that stalls past it gets badResumptionToken on the next request. The only safe recovery — shown in the loop above — is to discard the token and reissue the windowed ListRecords from the saved from cursor. Because every record write is idempotent, replaying the window’s earlier pages is a no-op, not duplication. Never fabricate or mutate a token, and never resume from a partial per-record cursor: both desynchronize the local catalog against the source.

5. Idempotent upsert per record identifier

Redelivery is guaranteed by design — window overlap, token restarts, and expiry replays all re-present records — so the write must converge. Key every upsert on the stable oai:identifier and treat a status="deleted" header as a withdrawal, not an insert.

def upsert(record_el):
    """Upsert one OAI record keyed on its identifier; honor deletion tombstones."""
    header = record_el.find("oai:header", OAI_NS)
    identifier = header.findtext("oai:identifier", namespaces=OAI_NS)
    if header.get("status") == "deleted":
        withdraw_from_index(identifier)          # tombstone -> remove, do not insert
        return
    body = record_el.find("oai:metadata", OAI_NS)
    store_record(identifier, etree.tostring(body))   # ON CONFLICT(identifier) DO UPDATE

The cursor store backing state.commit should be a durable table or key. A minimal, correct shape keyed per endpoint:

{
  "endpoint": "https://catalog.agency.gov/oai",
  "metadata_prefix": "iso19139",
  "last_ts": "2026-07-01T00:00:00Z",
  "token": null,
  "errors": 0
}

Tokens are the source’s state, not yours

A resumption token is an opaque handle to a cursor the server is holding, and every difficulty with paginated harvesting follows from that one fact. The token means nothing to the harvester, cannot be constructed or repaired locally, and — critically — may expire on the server’s schedule rather than yours. Treating it as a durable bookmark, to be stored and resumed hours later, is the single most common way a harvest of a large source fails halfway.

What a resumption token is, and what it is not Two columns listing what a token can be relied on for and what it cannot, with the operational consequence for a long traversal. Can rely on continuing this traversal, right now an ordering the server is maintaining a completeness count, when offered use it immediately; keep the loop tight Cannot rely on reusing it after a pause or a restart reconstructing it from anything local it surviving an upstream deployment never store it as a long-lived bookmark Consequence: a failure mid-traversal restarts the traversal — so the harvest must be idempotent, not resumable record progress by what was written, not by where the cursor was

The design that follows from this is to make restarting cheap rather than to make resuming reliable. Upsert every record as it arrives, so a restarted traversal re-writes rows that are already correct instead of duplicating them, and advance the persisted watermark only when a traversal completes. A partially completed harvest then costs bandwidth and time on the next attempt, and costs nothing in correctness — which is the right trade against a token that was never yours to keep.

Verification

Confirm the loop terminated cleanly, harvested every promised record, and is a no-op on re-run. Execute against a staging endpoint first.

# 1. The server's promised total for the window (completeListSize on the first token)
curl -s "https://catalog.agency.gov/oai?verb=ListRecords&metadataPrefix=iso19139&from=2026-07-01" \
  | xmllint --xpath 'string(//*[local-name()="resumptionToken"]/@completeListSize)' -
# -> e.g. 4820

# 2. The count actually stored after the harvest run
psql -At -c "SELECT count(*) FROM catalog_record WHERE harvested_at >= '2026-07-01';"
# -> 4820  (must equal completeListSize; a shortfall means a dropped page)

# 3. The cursor closed the window: token is NULL, watermark advanced
sqlite3 "$HARVEST_STATE_DB" "SELECT last_ts, token, errors FROM state;"
# -> 2026-07-13T00:00:00Z | (null) | 0

# 4. Re-running the same window is idempotent: row count does not change
python -m harvest.run --from 2026-07-01 --until 2026-07-02
psql -At -c "SELECT count(*) FROM catalog_record WHERE harvested_at >= '2026-07-01';"
# -> 4820  (unchanged: upserts collapsed the redelivered records)

A stored count equal to completeListSize, a null token with an advanced last_ts, and an unchanged row count on re-run together prove the paging loop is complete, crash-safe, and idempotent. Wire checks 1 and 2 into a pipeline gate so no promotion happens on a short harvest.

Guarding the loop against a source that never ends

A pagination loop trusts the server to stop it, and a misbehaving source will not. The failure modes are all variations on the same theme: a token that returns itself, a cursor that wraps to the beginning, a page count that never decreases. Each turns a harvest into an infinite loop that consumes the source, the harvester and the disk until somebody notices.

Three guards that bound a pagination loop Page ceiling, repeated-token detection and record ceiling, each with what it catches and the action it takes. page ceiling stop after N pages, where N exceeds any plausible traversal catches: a cursor that wraps stop and alert; never continue repeated token remember the tokens seen in this traversal catches: a token returning itself cheap: a set of hashes record ceiling stop above a multiple of the source's known size catches: unbounded duplication also protects the disk

All three guards should end the run and raise, not skip and carry on. A traversal that has started repeating is not producing useful data, and continuing past the guard both wastes the source’s capacity and buries the evidence of what went wrong under a large volume of duplicate records.

Troubleshooting matrix

Symptom Likely cause Fix
Loop never terminates resumptionToken element present-but-empty not treated as end-of-list Return None when the token text is empty or whitespace; break the loop
badArgument on the second page Token sent alongside from/metadataPrefix On any token request, send resumptionToken as the sole argument
badResumptionToken mid-window Token TTL exceeded while the worker stalled Discard the token, reissue windowed ListRecords from the saved from cursor
Restart skips a page Token persisted after processing instead of before Commit the token before upserting the page it belongs to
Stored count below completeListSize A page silently dropped, or until truncated the window Re-run the window from from; verify no exception swallowed a page
Duplicate rows after a crash Insert instead of upsert on the identifier Upsert keyed on oai:identifier; make replay a no-op
Deleted records reappear on re-harvest status="deleted" header inserted rather than withdrawn Branch on the header status; withdraw the identifier from the index

For the authoritative token and error-response semantics, the OAI-PMH 2.0 specification is definitive, and the lxml parsing documentation covers the production-safe parser tuning used above.

Up one level: Incremental Metadata Harvesting and Change Detection.