Geospatial Portal

Scoring Metadata Completeness in a CI Gate

This guide implements the scoring model and the blocking gate described in Metadata Quality Assurance & Record Scoring, and runs both in the ingestion pipeline. It sits within the Metadata Catalog Automation & Ingestion Workflows framework.

Prerequisites

  • Records available in a normalised internal form after the crosswalk, per mapping ISO 19115 to Dublin Core for CSW.
  • A CI runner able to execute the check against a harvest batch before it is published.
  • Provenance on every record — at minimum the source identifier — so the report can be grouped.
  • Agreement on the blocking rules with whoever will receive a rejection.

Separate the two outputs from the start

The check produces two things from one pass: a pass-or-fail verdict from a small set of blocking rules, and a score from everything else. Keeping them separate in the code is what keeps them separate in practice.

One pass, two outputs A record evaluated once, producing a blocking verdict and a weighted score with a per-rule breakdown. one record post-crosswalk form blocking rules four conditions, no threshold verdict: publish, or reject weighted rules a score plus a per-rule breakdown stored, never blocking the pipeline stops for this record, and the source is told which rule the report groups by source and rule, ordered by records a fix would improve

Step-by-step implementation

1. Express the rules as data, not as code branches 

# rules.py — every rule is a small callable with a weight and a plain-language
# description. Blocking rules carry weight 0 and are listed separately.
from dataclasses import dataclass
from typing import Callable

@dataclass(frozen=True)
class Rule:
    key: str
    describe: str          # written for a data custodian, not an engineer
    weight: int            # 0 for blocking rules
    check: Callable[[dict], bool]

def has_extent(r): 
    b = r.get("extent") or {}
    return all(isinstance(b.get(k), (int, float)) for k in ("minx", "miny", "maxx", "maxy")) \
        and b["maxx"] > b["minx"] and b["maxy"] > b["miny"]

BLOCKING = [
    Rule("identifier", "has a stable identifier we can update against", 0,
         lambda r: bool(r.get("identifier"))),
    Rule("extent_valid", "gives an area, so it can be found on a map", 0, has_extent),
    Rule("crs_known", "uses a coordinate reference system this portal publishes", 0,
         lambda r: r.get("crs") in PUBLISHED_CRS),
    Rule("date_parseable", "has a publication date we can read", 0,
         lambda r: r.get("published") is not None),
]

SCORED = [
    Rule("keywords", "has at least three keywords", 12,
         lambda r: len(r.get("keywords") or []) >= 3),
    Rule("abstract_len", "describes the data in more than a sentence", 14,
         lambda r: len((r.get("abstract") or "").split()) >= 40),
    Rule("lineage", "says where the data came from", 10,
         lambda r: bool(r.get("lineage"))),
    Rule("temporal", "states the period the data covers", 10,
         lambda r: bool(r.get("temporal_extent"))),
    Rule("contact", "names somebody who can answer questions about it", 12,
         lambda r: bool((r.get("contact") or {}).get("email"))),
    Rule("licence", "states what may be done with it", 12,
         lambda r: bool(r.get("licence"))),
    Rule("distribution", "offers at least one way to get the data", 18,
         lambda r: bool(r.get("distributions"))),
    Rule("format", "says what format the data is in", 6,
         lambda r: all(d.get("format") for d in (r.get("distributions") or []))),
    Rule("title_specific", "has a title that distinguishes it from its neighbours", 6,
         lambda r: len((r.get("title") or "")) >= 15),
]

2. Evaluate once, keep the breakdown 

# score.py — the breakdown is stored, not just the total, because that is what
# makes a disputed score a conversation rather than an argument.
def evaluate(record: dict) -> dict:
    blocked = [r.key for r in BLOCKING if not r.check(record)]
    results = {r.key: bool(r.check(record)) for r in SCORED}
    earned = sum(r.weight for r in SCORED if results[r.key])
    possible = sum(r.weight for r in SCORED)
    return {
        "identifier": record.get("identifier"),
        "source": record.get("provenance", {}).get("source_id"),
        "blocked_by": blocked,
        "score": round(100 * earned / possible),
        "rules": results,
        "model_version": MODEL_VERSION,     # so a trend break is explicable
    }

3. Run it in the pipeline and fail only on blocking rules 

# gate.py — exits non-zero only when a blocking rule failed. The score is
# always reported and never decides the exit code.
import json, sys, collections

def main(batch_path: str) -> int:
    results = [evaluate(r) for r in json.load(open(batch_path))]
    blocked = [x for x in results if x["blocked_by"]]

    by_source = collections.defaultdict(list)
    for x in results:
        by_source[x["source"]].append(x)

    print(f"{len(results)} records, {len(blocked)} blocked")
    for source, rows in sorted(by_source.items()):
        avg = round(sum(x["score"] for x in rows) / len(rows))
        print(f"  {source:32s} n={len(rows):5d} score={avg:3d}")

    json.dump(results, open("quality-report.json", "w"))
    return 1 if blocked else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))
# .github/workflows/harvest-quality.yml
- name: Metadata quality gate
  run: python3 gate.py harvest-batch.json
- name: Publish the report regardless of the verdict
  if: always()
  uses: actions/upload-artifact@v4
  with: {name: quality-report, path: quality-report.json}

4. Turn the report into work, ordered by leverage

Grouping decides whether the report is actionable A per-record listing beside a per-source-and-rule listing, with the number of records each fix would improve. grouped by record record 1 — no extent, no lineage record 2 — no extent, no licence record 3 — no extent, short abstract … 4,000 more lines nobody reads it, and no line suggests what to do first grouped by source and rule county-highways · no extent · 3,910 records district-planning · no licence · 640 records parish-portal · short abstract · 120 records the first line is one crosswalk fix that would improve nearly four thousand records at once

Tracking the trend without misreading it

A score is most useful as a series, and a series is easy to misread when the thing being measured changes underneath it. Three effects account for nearly every confusing movement.

Three movements that are not quality changes Model version, source composition and scored coverage, each with how it moves the average and what to do about it. the model changed a weight, or a new rule every record moves at once, in the same direction annotate the break; never compare across versions the population changed a large source onboarded the average moves while no record changed report per source as well as overall coverage changed more records now scored previously unscored records enter the average report the scored count alongside the score

Publish the scored count and the model version alongside every figure. It costs two extra columns and it removes most of the meetings that begin with somebody asking why the number moved.

Verification 

# 1. A record missing an extent is blocked, and the reason is named
echo '[{"identifier":"x","crs":"EPSG:27700","published":"2026-01-01"}]' > one.json
python3 gate.py one.json; echo "exit=$?"
#   expect: exit=1, and extent_valid listed in blocked_by

# 2. A thin but valid record is NOT blocked, and scores low
python3 - <<'EOF'
from score import evaluate
r = {"identifier":"y","crs":"EPSG:27700","published":"2026-01-01",
     "extent":{"minx":-1.4,"miny":50.8,"maxx":-1.2,"maxy":51.0},
     "title":"Parcels","abstract":"Parcels."}
out = evaluate(r); print(out["blocked_by"], out["score"])
EOF
#   expect: [] and a low score — reported, not rejected

# 3. The breakdown is stored per record
jq '.[0].rules' quality-report.json
#   expect: every scored rule with a boolean

# 4. The report groups by source
python3 gate.py harvest-batch.json | sed -n '2,10p'
#   expect: one line per source with a record count and an average

# 5. The model version travels with the score
jq -r '.[0].model_version' quality-report.json
#   expect: the current version — a trend break must be explicable

Troubleshooting matrix

Symptom Likely cause Fix
A whole source is blocked at once A blocking rule that is common upstream, not a data problem Move it to the score; blocking rules must be rare
Scores dropped sharply with no content change The model changed without a version bump Version the model; annotate the break rather than explaining it later
Publishers dispute the score and nothing improves Only the total is stored Store and expose the per-rule breakdown
The gate is slow on large harvests Rules re-parsing the record per check Evaluate once into a normalised dict; keep rules pure functions
Records pass but users cannot find them Extent validity checked as presence, not as usability Check the ordering and plausibility of the bounds, not just the fields
The report is ignored Grouped by record, and sent to a platform team Group by source and rule; send each source its own
Blocked records vanish silently The gate stops the batch without recording what was rejected Persist rejected records and their reasons for triage

FAQ

Should the gate fail the whole batch or only the affected records?

Only the affected records, with the rest published — otherwise one malformed record from one source stops an entire night’s harvest, and the pressure to weaken the rules becomes irresistible. Persist the rejected ones with their reasons so they can be triaged, and report the count per source.

What weight should each rule carry?

Something defensible rather than something precise. Weight by consequence for a user: a missing distribution link or extent costs more than a missing lineage statement, because the first two prevent use and the third merely reduces confidence. Write the reasoning down next to the weights so the next revision has something to argue with.

Should the score appear in the catalogue’s public interface?

A simplified version is useful — a small indicator that a record is thinly documented helps a user decide whether to rely on it and creates gentle pressure to improve. Publishing the raw number invites arguments about the model; naming what is missing is more useful and harder to dispute.

How do we avoid the score becoming a target?

By keeping it as a report rather than an objective, and by reviewing what it measures against what users need at least annually. A score that has become a target without being revisited is a good predictor of a catalogue where every abstract is exactly forty words long and none of them is useful.

Where should the rules themselves live?

In version control, next to the crosswalk they depend on, and reviewed like any other code. Rule changes have the same reach as mapping changes — one edit can move thousands of records — and a rule added to a running system without review is how a quality programme surprises its own operators.

Should the gate run on every harvest or only on new sources?

Every harvest, because records change and a source that was well-behaved last month can start emitting records without extents after a system upgrade on their side. The cost is negligible — the rules are pure functions over a parsed record — and running it only for new sources means the most common failure mode, a regression at an established source, is precisely the one nobody checks for.

Up one level: Metadata Quality Assurance & Record Scoring.