Building a GTFS Data Quality Scorecard

Normalise every validation finding into one record shape, scale each by the share of rows it affects rather than by its raw count, group them into a handful of dimensions an agency can actually act on, and combine those into one score with stated, versioned weights. Then store a scorecard per feed version, because the useful comparison is almost never against an absolute threshold — it is this feed against its own previous publication. The findings this consumes come from the checks described in error logging and data quality categorization.

Root Cause Analysis

A GTFS pipeline accumulates checks. Schema validation, referential integrity, calendar coverage, block plausibility, geometry sanity — each produces its own output in its own shape, and after a while nobody reads any of them. The report is thousands of lines, the important finding is on line 1,400, and an agency asking “is our feed any good” gets a log file.

Three things have to be true before an aggregate is worth computing.

Findings must be comparable. A validator that emits free text cannot be aggregated. Every check has to produce the same record: what it is, how severe, how many rows it touches, out of how many.

Counts must be normalised. Two hundred orphaned stop_times rows is fatal in a feed of a thousand and a rounding error in a feed of two million. Scoring on raw counts ranks agencies by size, not by quality — and because large agencies have more of everything, it ranks them worst.

Dimensions must map to owners. A single number says something is wrong. An agency needs to know whether the problem is in their schedule data, their geometry, their calendar or their identifiers, because those are maintained by different people and different systems.

There is also a failure specific to scoring: silently changing the weights. A score computed under one weighting and compared against one computed under another is meaningless, and the change is invisible unless the weight version is stored with the result.

Production-Ready Python Implementation

"""Aggregate GTFS validation findings into a per-publication scorecard."""
from __future__ import annotations

import logging
from collections import defaultdict
from dataclasses import dataclass, field, asdict
from datetime import date

log = logging.getLogger("gtfs.scorecard")

WEIGHTS_VERSION = "v2"       # bump whenever WEIGHTS or SEVERITY_COST changes

# The five things an agency can separately fix, and how much each matters.
WEIGHTS = {
    "integrity": 0.30,       # foreign keys, duplicate keys, orphaned rows
    "schedule": 0.25,        # times, sequences, trips that cannot run
    "calendar": 0.20,        # coverage, expiry, holiday modelling
    "geometry": 0.15,        # shapes, coordinates, snapping
    "metadata": 0.10,        # agency, timezone, versioning, fares
}

# How much a single finding costs, before it is scaled by how much it touches.
SEVERITY_COST = {"fatal": 1.0, "error": 0.6, "warning": 0.25, "info": 0.0}


@dataclass(frozen=True)
class Finding:
    check: str
    dimension: str
    severity: str
    affected_rows: int
    total_rows: int
    detail: str = ""

    @property
    def share(self) -> float:
        """How much of the relevant table this touches — never a raw count."""
        return min(1.0, self.affected_rows / self.total_rows) if self.total_rows else 0.0

    @property
    def cost(self) -> float:
        base = SEVERITY_COST.get(self.severity, 0.25)
        # A fatal finding costs its full weight however small its share: one trip
        # with no route is a defect, not a rounding error.
        return base if self.severity == "fatal" else base * self.share


@dataclass
class Scorecard:
    feed: str
    feed_version: str
    service_day: date
    dimensions: dict[str, float] = field(default_factory=dict)
    score: float = 0.0
    weights_version: str = WEIGHTS_VERSION
    findings: list[Finding] = field(default_factory=list)

    @property
    def worst_dimension(self) -> str | None:
        if not self.dimensions:
            return None
        return min(self.dimensions, key=self.dimensions.get)

    def as_row(self) -> dict:
        row = {k: v for k, v in asdict(self).items() if k != "findings"}
        row.update({f"dim_{k}": v for k, v in self.dimensions.items()})
        row.pop("dimensions")
        return row

    def explain(self) -> list[str]:
        """The findings that actually moved the score, worst first."""
        return [f"{f.dimension}/{f.check} [{f.severity}] {f.affected_rows}/"
                f"{f.total_rows} rows ({f.share:.1%}) {f.detail}"
                for f in sorted(self.findings, key=lambda f: f.cost, reverse=True)
                if f.cost > 0][:10]


def build_scorecard(feed: str, feed_version: str, service_day: date,
                    findings: list[Finding]) -> Scorecard:
    by_dimension: dict[str, list[Finding]] = defaultdict(list)
    for finding in findings:
        if finding.dimension not in WEIGHTS:
            log.error("finding %s names unknown dimension %r — not scored",
                      finding.check, finding.dimension)
            continue
        by_dimension[finding.dimension].append(finding)

    dimensions: dict[str, float] = {}
    for dimension in WEIGHTS:
        cost = sum(f.cost for f in by_dimension.get(dimension, ()))
        # A dimension bottoms out at zero rather than going negative, so one
        # catastrophic finding cannot drag the whole score below its own weight.
        dimensions[dimension] = max(0.0, 1.0 - cost)

    score = sum(WEIGHTS[d] * v for d, v in dimensions.items())

    card = Scorecard(feed=feed, feed_version=feed_version, service_day=service_day,
                     dimensions=dimensions, score=round(score, 4),
                     findings=findings)
    log.info("%s %s: score %.2f (%s) — weakest dimension %s at %.2f",
             feed, feed_version, card.score, WEIGHTS_VERSION,
             card.worst_dimension, dimensions.get(card.worst_dimension or "", 1.0))
    return card


def compare(previous: Scorecard, current: Scorecard,
            regression_threshold: float = 0.05) -> list[str]:
    """What got worse between two publications of the same feed."""
    if previous.weights_version != current.weights_version:
        return [f"scores are not comparable: weights {previous.weights_version} "
                f"vs {current.weights_version}"]

    regressions = []
    if current.score < previous.score - regression_threshold:
        regressions.append(
            f"overall score fell {previous.score:.2f} -> {current.score:.2f}")
    for dimension in WEIGHTS:
        before = previous.dimensions.get(dimension, 1.0)
        after = current.dimensions.get(dimension, 1.0)
        if after < before - regression_threshold:
            regressions.append(f"{dimension} fell {before:.2f} -> {after:.2f}")
    return regressions

Step-by-Step Walkthrough

Finding carries both the affected count and the total. That pair is what makes the share computable, and requiring both at construction means no check can contribute a raw count without saying what it is a count of. A check that cannot name its denominator has not thought about what it is measuring.

Fatal findings ignore their share. One trip referencing a route that does not exist is a defect whether the feed has a thousand trips or a million. Scaling it by 0.000001 would make a broken feed score perfectly. Every other severity scales, because for those the proportion genuinely is the story.

Dimension scores floor at zero. Without the floor, three fatal findings in one dimension would produce a negative dimension score that eats into the other dimensions’ contributions. Flooring keeps each dimension’s damage bounded by its own weight, so a feed with perfect geometry still scores at least 0.15 from geometry.

WEIGHTS_VERSION is stored on every scorecard. compare refuses to compare across versions rather than producing a difference that means nothing. This is the guard that keeps a year of history usable after someone decides calendar coverage should matter more.

explain returns only findings with non-zero cost, ranked. An info finding costs nothing and is therefore not why the score moved; including it in the explanation invites an agency to fix something that changes nothing.

compare reports per-dimension regressions, not just the total. A feed whose integrity improved and whose calendar collapsed can have an unchanged overall score. The dimension comparison is what surfaces that.

Verification and Output

def verify(card: Scorecard) -> None:
    assert abs(sum(WEIGHTS.values()) - 1.0) < 1e-9, "weights do not sum to 1"
    assert 0.0 <= card.score <= 1.0, f"score {card.score} outside the unit interval"
    assert set(card.dimensions) == set(WEIGHTS), "a dimension is missing from the card"
    for dimension, value in card.dimensions.items():
        assert 0.0 <= value <= 1.0, f"{dimension} scored {value}"

    recomputed = sum(WEIGHTS[d] * v for d, v in card.dimensions.items())
    assert abs(recomputed - card.score) < 1e-6, "score disagrees with its components"

    for finding in card.findings:
        assert finding.affected_rows >= 0
        assert finding.affected_rows <= finding.total_rows or finding.total_rows == 0, (
            f"{finding.check} affects more rows than exist")
        assert finding.severity in SEVERITY_COST, f"unknown severity {finding.severity}"

The recomputation assertion is the one that matters over time. It guarantees the stored score is exactly the weighted sum of the stored components, so a dashboard can display either and they will always agree.

A healthy feed:

INFO gtfs.scorecard: mbta 9f2a1c: score 0.97 (v2) — weakest dimension geometry at 0.88

A feed with a real problem:

INFO gtfs.scorecard: regional 4d81e0: score 0.61 (v2) — weakest dimension integrity at 0.00

integrity/stop_times_trip_fk [fatal] 41208/1840112 rows (2.2%) calls belonging to no trip
calendar/coverage_horizon [error] 1/1 rows (100.0%) 4 days of service remain
schedule/non_monotonic_times [warning] 1204/1840112 rows (0.1%) times decrease within a trip
geometry/missing_shape [warning] 2104/31042 rows (6.8%) trips with no shape_id

Four lines, ordered by how much each moved the score, each naming a table and a share. That is what an agency can act on, and the fact that integrity bottomed out at zero from a finding touching only 2.2% of rows is exactly the fatal-severity rule doing its job.

Across publications:

>>> compare(previous, current)
['overall score fell 0.94 -> 0.61', 'integrity fell 1.00 -> 0.00', 'calendar fell 0.95 -> 0.40']

Gotchas and Edge Cases

Checks that do not run. A feed with no shapes.txt produces no geometry findings, so geometry scores 1.0 — a perfect score for data that does not exist. Either emit an explicit info finding for a skipped check, or record coverage separately so a reader can tell “nothing wrong” from “nothing checked”.
Findings with a zero denominator. share returns 0.0, so a non-fatal finding against an empty table costs nothing. That is usually right, and it does mean an empty required table has to be reported as fatal to register at all.
Double counting. The same defect found by two checks contributes twice. Deduplicating findings by (dimension, check) before scoring is worth doing once a validation suite grows past a dozen checks.
Weight arguments. Every disagreement about a score is a disagreement about WEIGHTS. Keeping them in one dictionary with a version makes that a five-minute conversation instead of an archaeology exercise.
Scoring a feed nobody can fix. A third-party feed you merely consume gets the same score and no route to improvement. The scorecard is still worth computing — it tells you how much to trust the data — but framing it as a quality complaint to an agency that never sees it is wasted effort.

Frequently Asked Questions

Why weight by share of rows rather than by count?

Because a hundred orphaned stop times is catastrophic in a feed of a thousand rows and negligible in a feed of two million. Raw counts make large agencies look worse than small ones regardless of actual quality, which makes the score useless for comparison.

Should the score be shown to agencies?

Yes, but always with its components. A single number tells an agency that something is wrong; the dimension breakdown tells them which of five things to fix, and that is the only part that leads to a better feed.

How do I stop the score changing meaning over time?

Version the weights and store the version with every scorecard. Any change to the weighting makes historical scores incomparable, and without the version recorded nobody can tell which scores were computed under which rules.

What score is good enough?

There is no universal threshold. The useful test is relative: this feed against its own previous publication, and against the median of comparable feeds. A score that falls between publications is worth investigating whatever its absolute value.

Checking GTFS Referential Integrity in Python — the largest single source of findings
Detecting Duplicate Trips in GTFS Feeds — another check that feeds this aggregate
Up: Error Logging and Data Quality Categorization — the severity ladder the scoring rests on
Section: Python Parsing & Data Normalization · Home