Geospatial Portal

Detecting Broken Distribution Links in Catalog Records

This guide checks the links that make a catalogue record usable — downloads, service endpoints, documentation — on a schedule, and distinguishes a genuinely dead link from a server having a bad afternoon. It belongs to Metadata Quality Assurance & Record Scoring, within the Metadata Catalog Automation & Ingestion Workflows framework.

Prerequisites

  • Distribution links extracted into a queryable form, with their type and their owning record.
  • A scheduler able to run a bounded, polite crawl outside peak hours.
  • A per-source contact, so a persistent failure can be reported to somebody.
  • Agreement that a failing link marks a record rather than removing it.

The technical work here is trivial; the operational care is not. A catalogue of fifty thousand records may hold twice that many links, most of them pointing at other organisations’ servers, and an inconsiderate checker is indistinguishable from a denial-of-service attempt.

Four rules that keep a link check welcome Identification, per-host concurrency, request method, and scheduling, each with the failure it prevents. RULE WHY identify yourself an anonymous crawler gets blocked, and the block is permanent concurrency per host global limits still send one small server hundreds of requests at once HEAD, then ranged GET never download a multi-gigabyte file to prove it exists spread the schedule a catalogue's links cluster on a few hosts; check them over hours

Step-by-step implementation 

# linkcheck.py — the check depends on what the link is FOR.
import httpx

UA = "GeoportalLinkCheck/1.0 (+https://portal.example.gov/about/link-checking; ops@example.gov)"

async def check(client: httpx.AsyncClient, link: dict) -> dict:
    url, kind = link["url"], link["type"]

    # An OGC service endpoint is only usable if it answers a capabilities request.
    if kind in ("WMS", "WFS", "WCS", "CSW"):
        probe = f"{url}{'&' if '?' in url else '?'}service={kind}&request=GetCapabilities"
        r = await client.get(probe, timeout=20, follow_redirects=True)
        ok = r.status_code == 200 and b"<" in r.content[:200]
        return {"status": r.status_code, "ok": ok, "checked": "capabilities"}

    # A download: HEAD is enough, with a ranged GET fallback for servers that
    # refuse HEAD — which is common and is not a broken link.
    r = await client.head(url, timeout=20, follow_redirects=True)
    if r.status_code in (405, 501):
        r = await client.get(url, timeout=20, follow_redirects=True,
                             headers={"Range": "bytes=0-0"})
    return {"status": r.status_code, "ok": r.status_code < 400, "checked": "head"}

2. Bound the crawl per host 

import asyncio, collections
from urllib.parse import urlparse

HOST_LIMIT = 2          # simultaneous requests to any single host
HOST_DELAY = 1.0        # seconds between requests to the same host

_sem = collections.defaultdict(lambda: asyncio.Semaphore(HOST_LIMIT))

async def polite_check(client, link):
    host = urlparse(link["url"]).netloc
    async with _sem[host]:
        result = await check(client, link)
        await asyncio.sleep(HOST_DELAY)     # inside the semaphore, deliberately
        return result

A single failure is almost always noise: a restart, a certificate renewal, a network blip. Only a failure that persists across several checks on different days is evidence.

From one failure to a reported broken link Healthy, suspect, failing and broken states with the transitions between them and what each state triggers. healthy nothing reported suspect one failure — silent failing internal report only broken record marked, source told any success returns it to healthy immediately Requiring failures on different days, not merely consecutive checks, is what keeps a maintenance window from marking a source's whole catalogue.

4. Mark the record; do not remove it 

-- The record stays published, with an honest indication of the problem, and a
-- pointer to the last time the link was known to work.
ALTER TABLE distribution
  ADD COLUMN health text NOT NULL DEFAULT 'healthy',
  ADD COLUMN last_ok_at timestamptz,
  ADD COLUMN last_checked_at timestamptz,
  ADD COLUMN consecutive_failures int NOT NULL DEFAULT 0;

-- What the catalogue shows a user for a broken link.
SELECT r.title,
       d.url,
       d.health,
       d.last_ok_at
FROM record r JOIN distribution d USING (identifier)
WHERE d.health = 'broken';

Removing the link would be worse than marking it: a user who can see that a download stopped working on a known date can ask the custodian for it, whereas a record with no distribution at all looks like a dataset nobody ever published.

A catalogue record can carry several links, and treating them alike produces a report where a broken documentation URL sits next to an unavailable dataset. Weight the check by what the link is for.

Link types by consequence when they fail Distribution, documentation, publisher page and thumbnail, with the severity each failure deserves. download or service the dataset cannot be used — mark the record, tell the source documentation an inconvenience — report quietly, do not mark the record publisher page says nothing about the data — lowest priority thumbnail cosmetic — fix in bulk, never alert

Store the link type at ingestion rather than inferring it at check time from the URL, which is unreliable. The type comes from the source record and is exactly the kind of field a crosswalk quietly drops — worth asserting in the quality gate for that reason alone.

Verification 

# 1. A deliberately dead link is detected but not immediately reported
python3 linkcheck.py --only https://example.invalid/data.zip
psql -Atc "SELECT health, consecutive_failures FROM distribution WHERE url='https://example.invalid/data.zip';"
#   expect: suspect, 1

# 2. It becomes broken only after repeated failures on different days
for i in 1 2 3; do python3 linkcheck.py --only https://example.invalid/data.zip --simulate-day $i; done
psql -Atc "SELECT health FROM distribution WHERE url='https://example.invalid/data.zip';"
#   expect: broken

# 3. A recovery clears it immediately
python3 linkcheck.py --only https://portal.example.gov/downloads/parcels.zip
psql -Atc "SELECT health, last_ok_at IS NOT NULL FROM distribution WHERE url LIKE '%parcels.zip';"
#   expect: healthy, t

# 4. No host received more than the configured concurrency
awk '{print $3}' linkcheck.log | sort | uniq -c | sort -rn | head -5
#   cross-check the timestamps: no more than two in flight per host

# 5. Service endpoints are checked as services, not as URLs
grep '"checked":"capabilities"' linkcheck-results.json | wc -l
#   expect: one per OGC service link, not a plain HEAD

Check 5 matters because an OGC endpoint that returns HTTP 200 for a bare URL and an error for a capabilities request is broken in the way that affects users, and a naive checker calls it healthy.

Troubleshooting matrix

Symptom Likely cause Fix
A partner blocks the checker No identification, or too much concurrency per host Identify in the user agent; limit per host and add a delay
Everything from one source fails at once That source had a maintenance window Require failures across different days before reporting
Links reported healthy but users cannot download HEAD accepted while the GET fails, or a login page returns 200 Check the content type and size, not only the status code
Service endpoints marked healthy while broken Checked as a plain URL rather than with a capabilities request Type the link and check each kind appropriately
The check takes longer than its window Per-host delays serialised across a few dominant hosts Interleave hosts; schedule the crawl over hours, not minutes
Redirect chains reported as failures Redirects not followed, or a loop Follow a bounded number of redirects; report loops distinctly
Records disappear when a link breaks Removal rather than marking Mark and keep, with the date the link last worked

FAQ

Weekly for the whole catalogue is a reasonable default, with more frequent checks for the small number of links that are heavily used. The cadence matters less than the persistence rule — checking daily and reporting on the first failure produces more noise than checking weekly and reporting on the third.

Yes, and it should be visible separately from ingestion-time quality, because the two have different owners. A record that arrived without a link is a publisher issue; a record whose link rotted is usually somebody else’s change, and conflating them makes both reports harder to act on.

Check what can be checked — that the host resolves and returns an authentication challenge rather than a connection failure — and mark the rest as unverifiable rather than as healthy. Recording “we cannot check this” honestly is more useful than a green tick that means nothing.

Yes, briefly and factually: an indication that a download has not worked since a given date, alongside the custodian’s contact. It saves the user the attempt, gives them the route to ask, and creates the pressure that gets it fixed.

Exclude the portal’s own hosts from external reporting and monitor them through the normal availability alerting instead. A catalogue that reports its own scheduled maintenance as thousands of broken links teaches everyone to ignore the report.

Escalate it to a curation decision rather than leaving it marked indefinitely. Either the custodian can supply a new location, or the distribution should be removed from the record with a note, or the record itself is obsolete and should be retired — all three are legitimate, and none of them is “leave it marked broken for another year”.

No. A ranged request that asks for the first byte establishes that the resource exists and is being served, which is what the check needs, and it avoids transferring gigabytes per record per week. Where a server refuses ranged requests, record the link as unverifiable rather than downloading it — an honest gap is better than a check that costs the publisher real bandwidth every week.

Twenty seconds is generous enough for a slow but working server and short enough that a whole catalogue’s crawl finishes inside its window. Distinguish a timeout from a refusal in the record, because they mean different things: a timeout usually indicates an overloaded server that will recover, while a connection refusal or a 404 is far more likely to be permanent.

Up one level: Metadata Quality Assurance & Record Scoring.