Tracking trip_id Match Rates in Python

Build a set of trip identifiers from the static feed you currently have loaded, resolve every realtime entity’s TripDescriptor against it, and — this is the part that makes the metric useful — classify every miss by cause rather than lumping them into one number. Then alert on a step change against a rolling baseline rather than on an absolute threshold, because every agency’s steady-state rate is different and only movement is diagnostic. This is the second of the metrics in monitoring realtime feed quality.

Root Cause Analysis

The realtime and static halves of GTFS are published independently, versioned independently, and joined by identifiers neither side guarantees to keep stable. Everything a consumer does with realtime data depends on that join succeeding: without a matched trip there is no scheduled time to compare a delay against, no stop sequence to attach a prediction to, and no route to label a vehicle with. An unmatched entity is not degraded data, it is data that has to be discarded.

The join fails for four distinct reasons, and they are routinely reported as one.

Version drift. The agency republishes the static feed and identifiers change. The realtime producer catches up within a cycle or two; until it does, a large share of entities reference trips that no longer exist. This is the cause of nearly every sudden drop, and it resolves itself — which is why alerting on it as an incident annoys people, and why not noticing it at all means a genuine, non-recovering drift looks the same.

No descriptor at all. Some producers publish vehicle positions with a vehicle identifier and no trip_id, especially for vehicles between assignments. These cannot be matched by definition and should not count against the match rate the same way a broken reference does.

A trip that exists but is not running today. The trip_id resolves to a row in trips.txt, but its service_id is not active on the current service date. The join technically succeeds and the result is still wrong, because the entity claims a trip that today’s service calendar says is not operating. This one is invisible unless you check for it deliberately.

Deadheading and out-of-service vehicles. Real, expected, and correctly unmatched. A monitoring system that treats these as failures will report a permanent 3–5% miss rate that nobody can ever fix.

Production-Ready Python Implementation

"""Measure and diagnose the GTFS-Realtime to static trip join."""
from __future__ import annotations

import logging
import statistics
from collections import Counter, deque
from dataclasses import dataclass
from datetime import date

log = logging.getLogger("gtfs.rt.match")

BASELINE_POLLS = 60            # roughly half an hour at a 30-second cadence
STEP_DROP = 0.10               # a fall this far below baseline is a step change
MIN_ENTITIES = 25              # below this, the rate is too noisy to judge

# Miss causes, in the order they are tested.
NO_DESCRIPTOR = "no trip_id in the descriptor"
UNKNOWN_TRIP = "trip_id not in the static feed"
NOT_SCHEDULED_TODAY = "trip exists but its service is not active today"
ADDED_TRIP = "trip added by the realtime feed (schedule_relationship ADDED)"


@dataclass(frozen=True)
class MatchResult:
    entities: int
    matched: int
    causes: Counter

    @property
    def rate(self) -> float:
        return self.matched / self.entities if self.entities else 0.0

    @property
    def matchable(self) -> int:
        """Entities that could in principle have matched — excludes ADDED and
        descriptor-less vehicles, which are not defects."""
        excused = self.causes[NO_DESCRIPTOR] + self.causes[ADDED_TRIP]
        return self.entities - excused

    @property
    def strict_rate(self) -> float:
        return self.matched / self.matchable if self.matchable else 0.0


class MatchMonitor:
    def __init__(self, feed_name: str):
        self.feed_name = feed_name
        self._history: deque[float] = deque(maxlen=BASELINE_POLLS)

    def evaluate(self, message, static_trips: dict[str, str],
                 active_services: set[str], service_day: date) -> MatchResult:
        """static_trips: trip_id -> service_id. active_services: service_ids running today."""
        causes: Counter = Counter()
        matched = entities = 0

        for entity in message.entity:
            descriptor = None
            if entity.HasField("vehicle") and entity.vehicle.HasField("trip"):
                descriptor = entity.vehicle.trip
            elif entity.HasField("trip_update"):
                descriptor = entity.trip_update.trip
            else:
                continue                       # alerts carry no trip descriptor
            entities += 1

            relationship = descriptor.schedule_relationship
            if relationship == 1:              # ADDED — no static counterpart exists
                causes[ADDED_TRIP] += 1
                continue

            trip_id = descriptor.trip_id
            if not trip_id:
                causes[NO_DESCRIPTOR] += 1
                continue

            service_id = static_trips.get(trip_id)
            if service_id is None:
                causes[UNKNOWN_TRIP] += 1
                continue
            if service_id not in active_services:
                causes[NOT_SCHEDULED_TODAY] += 1
                continue
            matched += 1

        result = MatchResult(entities=entities, matched=matched, causes=causes)
        if entities >= MIN_ENTITIES:
            self._history.append(result.strict_rate)
        return result

    def baseline(self) -> float | None:
        if len(self._history) < 10:
            return None
        return statistics.median(self._history)

    def assess(self, result: MatchResult) -> list[str]:
        if result.entities < MIN_ENTITIES:
            return []                          # too few entities to say anything

        problems = []
        base = self.baseline()
        if base is not None and result.strict_rate < base - STEP_DROP:
            dominant = result.causes.most_common(1)
            detail = f"; mostly {dominant[0][0]}" if dominant else ""
            problems.append(
                f"match rate stepped down: {result.strict_rate:.0%} against a "
                f"{base:.0%} baseline{detail}")

        if result.causes[UNKNOWN_TRIP] > result.entities * 0.25:
            problems.append(
                f"{result.causes[UNKNOWN_TRIP]} of {result.entities} entities "
                "reference trips absent from the static feed — the static feed has "
                "probably been republished")

        if result.causes[NOT_SCHEDULED_TODAY] > result.entities * 0.10:
            problems.append(
                f"{result.causes[NOT_SCHEDULED_TODAY]} entity(s) name trips that are "
                "not scheduled today — check the service-date resolution, especially "
                "either side of midnight")
        return problems

Step-by-Step Walkthrough

Alerts are skipped entirely. A ServiceAlert entity carries no TripDescriptor, so counting it as an unmatched entity would drag the rate down by however many disruptions the agency currently has published. The continue before entities += 1 keeps them out of the denominator.

ADDED trips are excused, not counted as misses. schedule_relationship = ADDED means the producer is announcing a trip that is deliberately not in the static feed — an extra service laid on for an event. It cannot match, and it is not a defect. Counting it as one gives a permanently depressed rate on any agency that runs extras.

Two rates are exposed. rate is over everything; strict_rate excludes the excused categories. The strict rate is what should be monitored, because it moves only when something is actually wrong. The raw rate is still useful for reporting how much of the feed you are able to use.

The NOT_SCHEDULED_TODAY check is the one nobody writes. The trip exists, the join succeeds, and the entity is still wrong — because that trip’s service is not running today. It is nearly always a service-date bug: a vehicle running at 00:30 belongs to the previous service day, and resolving it against today’s calendar makes every overnight trip look unscheduled. That is the same service-day boundary that catches out schedule normalisation.

The baseline is a median over recent polls, not a configured constant. Every agency has a different steady state, and a threshold that works for one is wrong for the next. Using the feed’s own recent history means the check needs no per-agency tuning and adapts when a feed genuinely improves.

Low-entity polls are excluded from the baseline and from assessment. At 03:00 there may be four vehicles running, and one unmatched entity is then a 25% miss rate. MIN_ENTITIES keeps that noise out of both the history and the alerts.

Verification and Output

def verify(result: MatchResult) -> None:
    accounted = result.matched + sum(result.causes.values())
    assert accounted == result.entities, (
        f"{result.entities} entities but {accounted} accounted for — a code path "
        "neither matches nor records a cause")
    assert 0.0 <= result.rate <= 1.0
    assert result.matchable <= result.entities
    assert result.strict_rate >= result.rate - 1e-9, (
        "the strict rate must be at least the raw rate, since it excludes only "
        "entities that could never have matched")

The first assertion is the one that catches real bugs. Every entity must either match or produce exactly one cause; a continue added later without a corresponding counter increment silently removes entities from the accounting, and the rate then drifts without any visible reason.

Steady state:

INFO gtfs.rt.match: mbta_trip_updates: 1284 entities, 1247 matched (97%), strict 99%, causes: {'trip added by the realtime feed (schedule_relationship ADDED)': 21, 'trip_id not in the static feed': 12, 'trip exists but its service is not active today': 4}

A static republication in progress:

WARN gtfs.rt.match: mbta_trip_updates: match rate stepped down: 68% against a 99% baseline; mostly trip_id not in the static feed
WARN gtfs.rt.match: 402 of 1290 entities reference trips absent from the static feed — the static feed has probably been republished

And the service-date bug, which looks nothing like the others:

WARN gtfs.rt.match: night_network: 61 entity(s) name trips that are not scheduled today — check the service-date resolution, especially either side of midnight

Gotchas and Edge Cases

• Reloading the static feed mid-poll. If the static trip set is swapped while a message is being evaluated, some entities are matched against the old feed and some against the new. Pin the static snapshot for the duration of each evaluation, as the matching pipeline does.
• Producers that populate trip_id inconsistently. Some feeds supply it on TripUpdate and omit it on VehiclePosition, so the two entity types have very different match rates. Track them separately where the difference is large, or the combined figure describes neither.
• schedule_relationship values beyond ADDED. CANCELED trips still reference a real static trip and should match normally; UNSCHEDULED behaves like ADDED. Test the enum explicitly rather than assuming anything non-zero is exceptional.
• A baseline poisoned by a long outage. If the rate sits at 40% for an hour, the median baseline drifts down and the recovery to 99% is never flagged as unusual — which is correct — but a subsequent drop to 60% is then compared against a contaminated baseline. Freezing the baseline while an alert is active avoids this.
• Feeds with very small fleets. A rural operator with eight vehicles never reaches MIN_ENTITIES, so nothing is ever assessed. Lower the threshold per feed rather than removing it, and accept that the rate will be noisy.

Frequently Asked Questions

Related

• Matching Realtime to Static Schedules — the join whose success this measures
• Measuring GTFS-Realtime Feed Staleness — the other metric that moves when a producer changes
• Up: Monitoring Realtime Feed Quality — where match rate sits among the four metrics
• Section: GTFS-Realtime Integration · Home