Monitoring Realtime Feed Quality

A GTFS-Realtime feed fails far more often by going quiet than by going down. The endpoint keeps answering, the protobuf keeps decoding, and the data stops changing — and every consumer downstream keeps serving predictions built on a snapshot that is now half an hour old. HTTP monitoring cannot see this, because nothing about the request failed. The only signals that can are inside the FeedMessage itself and in its relationship to the static schedule.

This page covers the four measurements worth taking on every poll: how old the data is, how much of it resolves against the static feed, how much of the scheduled service it covers, and whether the sequence of successful polls has any holes in it. It also covers the part that decides whether anyone acts on them — alerting that understands a network which does not run at three in the morning.

Prerequisites

• A working GTFS-Realtime consumer; the metrics here are computed from the FeedMessage you already decode
• The current static feed loaded, with the service calendar expanded so “what should be running now” is answerable
• Somewhere to keep a short history of poll results — a few hours in memory is enough for every metric below

pip install gtfs-realtime-bindings pandas

Concept and Spec Background

The header timestamp is the only honest clock

Every FeedMessage carries a header.timestamp in POSIX seconds UTC: the moment the producer generated the message. It is not the moment you fetched it, and it is not the moment any individual entity was observed. A feed that is frozen serves the same header timestamp indefinitely while your own clock advances, and the difference between the two is the single most valuable number in real-time monitoring.

header.incrementality matters here too. FULL_DATASET means the message replaces everything you hold; DIFFERENTIAL means it patches it. The overwhelming majority of published feeds are FULL_DATASET, and a consumer that assumes differential semantics on a full feed will accumulate entities that should have been dropped.

Per-entity timestamps are the second layer. A feed can have a fresh header and entities that have not moved in twenty minutes, which is the signature of an upstream AVL system failing while the feed generator carries on.

Four metrics, and what each catches

Staleness is now - header.timestamp. It catches the frozen feed, which is the most common failure and the one HTTP checks miss entirely.

Match rate is the share of entities whose trip_id resolves to a trip in the current static feed. It catches version drift — the agency republished the static feed, identifiers changed, and the two halves no longer agree.

Coverage is the share of scheduled trips the feed says anything about. It catches a feed that is fresh, matches perfectly, and only reports on a quarter of the fleet because one garage’s AVL is offline.

Continuity is whether the sequence of successful polls has holes. It catches intermittent failure, which no single poll can see.

The four are independent. A feed can score perfectly on three and fail the fourth, and each failure has a different owner: staleness and continuity belong to the publisher’s infrastructure, match rate to whoever republished the static feed, coverage to the agency’s own vehicle systems.

Step-by-Step Implementation

Step 1 — Capture a poll result

from dataclasses import dataclass, field
from datetime import datetime, timezone


@dataclass(frozen=True)
class PollResult:
    fetched_at: float                 # POSIX seconds, our clock
    header_timestamp: int             # POSIX seconds, the producer's clock
    entity_count: int
    matched_count: int
    scheduled_now: int                # trips the static schedule says are running
    ok: bool = True
    error: str | None = None

    @property
    def staleness_s(self) -> float:
        return self.fetched_at - self.header_timestamp

    @property
    def match_rate(self) -> float:
        return self.matched_count / self.entity_count if self.entity_count else 0.0

    @property
    def coverage(self) -> float:
        return self.matched_count / self.scheduled_now if self.scheduled_now else 0.0

Every metric is a property derived from four recorded numbers. Storing derived values instead invites two of them to disagree after a bug fix.

Step 2 — Derive the cadence rather than assuming it

import statistics


def observed_cadence_s(history: list[PollResult]) -> float | None:
    """How often the producer actually publishes, from distinct header timestamps."""
    stamps = sorted({p.header_timestamp for p in history if p.ok})
    if len(stamps) < 4:
        return None
    gaps = [b - a for a, b in zip(stamps, stamps[1:])]
    return statistics.median(gaps)

A hard-coded staleness threshold is wrong for most feeds. Agencies publish anywhere from every five seconds to every five minutes, and the correct threshold is a multiple of whatever this particular producer does. Deduplicating the timestamps first is essential: polling faster than the producer publishes returns the same message repeatedly, and counting those as separate observations would report a cadence of zero.

Step 3 — Judge a single poll

STALENESS_MULTIPLIER = 3.0
MIN_MATCH_RATE = 0.90
MIN_COVERAGE = 0.60


def judge(poll: PollResult, cadence_s: float | None) -> list[str]:
    problems = []
    if not poll.ok:
        return [f"poll failed: {poll.error}"]

    threshold = (cadence_s or 60.0) * STALENESS_MULTIPLIER
    if poll.staleness_s > threshold:
        problems.append(
            f"stale: header is {poll.staleness_s:.0f}s old against a "
            f"{cadence_s or 60:.0f}s cadence")

    if poll.entity_count and poll.match_rate < MIN_MATCH_RATE:
        problems.append(
            f"match rate {poll.match_rate:.0%} — the static feed may have moved "
            "underneath the realtime one")

    # Coverage is only meaningful when something is scheduled to be running.
    if poll.scheduled_now > 0 and poll.coverage < MIN_COVERAGE:
        problems.append(
            f"coverage {poll.coverage:.0%}: {poll.matched_count} entities against "
            f"{poll.scheduled_now} scheduled trip(s)")
    return problems

The scheduled_now > 0 guard is what makes overnight quiet periods silent. At 03:00 on a network that does not run overnight, scheduled_now is zero, coverage is undefined rather than terrible, and nobody is paged.

Step 4 — Alert on sustained degradation

SUSTAINED_POLLS = 5


def sustained_problems(history: list[PollResult], cadence_s: float | None) -> list[str]:
    """Only problems present in every one of the last N polls are worth waking anyone."""
    recent = history[-SUSTAINED_POLLS:]
    if len(recent) < SUSTAINED_POLLS:
        return []
    sets = [set(judge(p, cadence_s)) for p in recent]
    persistent = set.intersection(*sets) if sets else set()
    return sorted(persistent)

A single bad poll is noise: a dropped packet, a producer restart, a garbage-collection pause. A problem present in five consecutive polls is a fact. Intersecting the problem sets rather than counting occurrences also means a feed alternating between two different faults does not silently pass.

Validation and Verification

def verify(history: list[PollResult]) -> None:
    for poll in history:
        assert poll.matched_count <= poll.entity_count, "more matches than entities"
        assert 0.0 <= poll.match_rate <= 1.0
        assert poll.header_timestamp > 0 or not poll.ok, "healthy poll with no header"
        # A negative staleness means the producer's clock is ahead of ours.
        if poll.ok and poll.staleness_s < -5:
            raise AssertionError(
                f"header timestamp is {-poll.staleness_s:.0f}s in the future — "
                "check clock sync on both ends before trusting any staleness figure")

    stamps = [p.header_timestamp for p in history if p.ok]
    assert stamps == sorted(stamps), "header timestamps went backwards"

The clock-skew assertion earns its place. A monitoring host whose clock drifts makes every staleness figure wrong in a direction nobody notices, because the number still looks plausible. A header timestamp in the future is impossible and points straight at the cause.

The backwards-timestamp check catches a load-balanced producer serving from two nodes with different states — a real failure mode that makes predictions oscillate and is otherwise very hard to diagnose.

Failure Modes and Edge Cases

• A feed that is fresh but empty. Zero entities with a moving header timestamp, during service hours. The producer is healthy and has nothing to say, which means the upstream vehicle system is down. Coverage catches it; staleness never will.
• A match rate that drops in one step. Almost always the static feed being republished with new identifiers. It resolves itself when the realtime side catches up, which is why alerting on sustained degradation rather than on a single poll matters.
• DIFFERENTIAL incrementality. Rare, and it breaks the entity count as a measure of fleet size, because each message carries only what changed. Detect the field and measure differently rather than reporting a coverage collapse.
• Duplicate entity ids within one message. Invalid, and it inflates entity_count so match rate looks worse than it is. Deduplicate on entity.id before counting.
• Per-entity timestamps that never move. A fresh header hiding a stale fleet. Worth a fifth metric on feeds that populate vehicle.timestamp: the median age of individual entities, which should track the header closely.
• Scheduled trips that genuinely have no vehicle. Some agencies only equip part of their fleet. Coverage will sit permanently at, say, 65%, and the threshold has to be set from that agency’s observed baseline rather than from a universal number.

Performance and Scale Notes

None of these metrics costs anything meaningful. Staleness is one subtraction. Match rate needs a set membership test per entity against a set of trip identifiers already in memory from the static feed — for 4,000 entities that is well under a millisecond. Coverage needs the count of trips scheduled to be running at the current moment, which comes from the same expanded calendar and stop-time index the consumer already builds.

The one thing to watch is the history buffer. Keeping every poll for a 15-second feed is 5,760 records a day per feed, which is trivial for one feed and less trivial across a hundred agencies. Keep a bounded deque of the last few hundred polls in memory for alerting, and write the per-poll record to storage for trend analysis rather than holding it. The daily aggregate — median staleness, minimum match rate, hours below coverage threshold — is what tells you whether a feed is getting better or worse, and that belongs in the same partitioned store as everything else historical.

The static side deserves a note of its own. Match rate and coverage both need the current static feed indexed by trip identifier and by active service, and rebuilding those indexes on every poll would dominate the cost of monitoring entirely. Build them once when a static feed version is loaded, hold them behind the feed’s content checksum, and swap the whole snapshot atomically when the agency republishes. Swapping it field by field, or rebuilding it lazily on first miss, produces a window during which entities are matched against a half-updated index — which registers as a match-rate collapse that has no cause anyone can find afterwards, because by the time someone looks the index is consistent again.

One further economy is worth taking. The static index a monitor needs is small — a set of trip identifiers and a set of active service identifiers — so several feeds belonging to the same publisher can share one snapshot rather than each holding a copy. Where a publisher issues one static feed and three realtime endpoints, that turns three index builds per republication into one, and it removes the possibility of the three monitors disagreeing about which static version they are measuring against.

Across many agencies, poll and judge independently per feed. One agency’s outage must not delay another’s monitoring, which is the same isolation principle that governs multi-agency batch processing.

Frequently Asked Questions

Related

• Measuring GTFS-Realtime Feed Staleness — the header clock, cadence derivation and clock-skew traps in full
• Tracking trip_id Match Rates in Python — measuring and diagnosing the realtime-to-static join
• Alerting on Realtime Feed Outages — schedule-aware alerting that survives the night shift
• Building a Realtime Feed Quality Dashboard — turning the metrics into something an operations team reads
• Matching Realtime to Static Schedules — the join whose success rate this section measures

Up: GTFS-Realtime Integration | Home