1. Home
  2. GTFS Realtime Integration
  3. Monitoring Realtime Feed Quality
  4. Building A Realtime Feed Quality Dashboard

Building a Realtime Feed Quality Dashboard

Aggregate the per-poll metrics into one record per feed per service day, using percentiles for staleness rather than a mean, and weighting coverage by how much service was actually scheduled in each hour. Then combine them into a single comparable score while keeping every component visible, because the score is what makes a hundred feeds rankable and the components are what tell anyone what to do. This is the reporting layer over the metrics defined in monitoring realtime feed quality.

Why the mean hides the outage Staleness statistics for a day containing a 45-minute freeze: only the tail moves, which is the part riders experienced. Mean staleness 51 s barely moved by 90 bad samples Median 38 s unaffected 95th percentile 1802 s the outage, plainly Maximum 2704 s report the tail; the mean describes a day that did not happen

Root Cause Analysis

Per-poll metrics answer “is this feed working right now”. They are the wrong instrument for three questions an operations team asks constantly: is this feed getting worse, is it worse than the others, and was yesterday unusual. Answering those needs aggregation, and aggregation is where most feed dashboards go wrong.

The mean hides the outage. A feed polled every 30 seconds produces 2,880 samples a day. If it is frozen for 45 minutes, that is 90 bad samples out of 2,880 — the mean staleness barely moves, and the dashboard shows a healthy day. The 95th percentile moves sharply, because the tail is exactly where the outage lives.

Unweighted hourly averages flatter a broken peak. Averaging 24 hourly coverage figures gives 03:00 — when two trips are scheduled — the same weight as 08:00, when four hundred are. A feed that collapses entirely during the morning peak and recovers by ten o’clock scores well. Weighting by scheduled trips makes the number mean what people assume it means.

Feeds are not comparable without normalisation. One agency publishes every 5 seconds, another every 5 minutes; one runs 20 hours a day, another 6. Raw staleness seconds and raw entity counts cannot be ranked across them. Every component has to be scaled against what is normal for that feed before a cross-agency view means anything.

And one thing that is not a modelling problem but ruins dashboards anyway: aggregating by calendar day rather than by service day. A network running until 01:30 has its late-night performance attributed to the following day, so a bad Friday night shows up as a bad Saturday morning, and nobody can correlate it with anything.

Production-Ready Python Implementation

python 
"""Aggregate GTFS-Realtime poll metrics into a daily feed quality record."""
from __future__ import annotations

import logging
import statistics
from collections import defaultdict
from dataclasses import dataclass, asdict
from datetime import date

log = logging.getLogger("gtfs.rt.dashboard")

# Component weights. They sum to 1.0 and are stated here rather than buried
# in the arithmetic, because every argument about the score is an argument
# about these three numbers.
WEIGHTS = {"freshness": 0.4, "match": 0.35, "coverage": 0.25}

# A feed is "fresh" while staleness stays under this many publication intervals.
FRESH_MULTIPLIER = 2.0


@dataclass(frozen=True)
class PollSample:
    service_day: date
    hour: int                   # hour of the service day, may exceed 23
    staleness_s: float
    match_rate: float
    matched: int
    scheduled_now: int
    ok: bool


@dataclass
class DailyQuality:
    feed: str
    service_day: date
    polls: int
    failed_polls: int
    cadence_s: float
    staleness_p50: float
    staleness_p95: float
    staleness_max: float
    minutes_stale: float
    match_rate_p05: float       # the bad end of the distribution, not the middle
    match_rate_median: float
    coverage_weighted: float
    score: float

    def as_row(self) -> dict:
        return asdict(self)


def _percentile(values: list[float], q: float) -> float:
    if not values:
        return 0.0
    ordered = sorted(values)
    index = min(len(ordered) - 1, max(0, int(round(q * (len(ordered) - 1)))))
    return ordered[index]


def summarise_day(feed: str, samples: list[PollSample], cadence_s: float) -> DailyQuality:
    if not samples:
        raise ValueError(f"{feed}: no samples for this service day")

    good = [s for s in samples if s.ok]
    failed = len(samples) - len(good)

    staleness = [s.staleness_s for s in good]
    fresh_limit = cadence_s * FRESH_MULTIPLIER
    stale_polls = sum(1 for v in staleness if v > fresh_limit)
    minutes_stale = stale_polls * cadence_s / 60.0

    match_rates = [s.match_rate for s in good if s.scheduled_now > 0]

    # Coverage weighted by scheduled service: an hour with two scheduled trips
    # must not count the same as the morning peak.
    by_hour_matched: dict[int, int] = defaultdict(int)
    by_hour_scheduled: dict[int, int] = defaultdict(int)
    for s in good:
        by_hour_matched[s.hour] += s.matched
        by_hour_scheduled[s.hour] += s.scheduled_now
    total_scheduled = sum(by_hour_scheduled.values())
    coverage = (sum(min(by_hour_matched[h], by_hour_scheduled[h])
                    for h in by_hour_scheduled) / total_scheduled
                if total_scheduled else 0.0)

    freshness_component = 1.0 - (stale_polls / len(good)) if good else 0.0
    match_component = statistics.median(match_rates) if match_rates else 0.0
    score = (WEIGHTS["freshness"] * freshness_component
             + WEIGHTS["match"] * match_component
             + WEIGHTS["coverage"] * coverage)

    quality = DailyQuality(
        feed=feed,
        service_day=samples[0].service_day,
        polls=len(samples),
        failed_polls=failed,
        cadence_s=cadence_s,
        staleness_p50=_percentile(staleness, 0.50),
        staleness_p95=_percentile(staleness, 0.95),
        staleness_max=max(staleness) if staleness else 0.0,
        minutes_stale=minutes_stale,
        match_rate_p05=_percentile(match_rates, 0.05),
        match_rate_median=match_component,
        coverage_weighted=coverage,
        score=round(score, 4),
    )
    log.info("%s %s: score %.2f (fresh %.2f, match %.2f, coverage %.2f), "
             "p95 staleness %.0fs, %.0f min stale, %d failed poll(s)",
             feed, quality.service_day, quality.score, freshness_component,
             match_component, coverage, quality.staleness_p95,
             quality.minutes_stale, failed)
    return quality


def to_frame(records: list[DailyQuality]):
    import pandas as pd
    return pd.DataFrame([r.as_row() for r in records])
From poll to a row anyone reads Per-poll samples are aggregated per service day, weighted by scheduled service, and combined into one score whose components stay visible. Samples one per poll Aggregate per service day Weight by scheduled service Score components kept a dashboard showing only a score sends people back to the raw logs

Step-by-Step Walkthrough

Samples carry a service_day, not a calendar date. The caller resolves it once, when the poll is recorded, using the same service-day rule the static schedule uses. A 01:15 poll on a network running past midnight belongs to the previous service day, and attributing it correctly is what lets a bad Friday night appear on Friday.

hour may exceed 23, for the same reason. Hour 25 of the service day is 01:00 the next morning, and keeping it in service-day space means the hourly coverage weighting lines up with the schedule that produced scheduled_now.

Freshness is a share of polls, not a duration. 1 - stale_polls / polls is bounded in [0, 1] and therefore combines cleanly with the other two components. minutes_stale is reported alongside it as the human-readable version, because “47 minutes stale” is what an operations team can act on while “freshness 0.97” is not.

The match component uses the median, but the 5th percentile is reported too. The median describes the normal day; the 5th percentile describes the worst part of it. A feed with a median of 0.99 and a 5th percentile of 0.42 had a bad half hour, which the median alone conceals.

Coverage is a weighted ratio, not a mean of ratios. Summing matched and scheduled counts across the whole day and dividing once weights every hour by how much service it carried. min(matched, scheduled) guards the case where the feed reports more entities than the schedule expects — extras, added trips — which would otherwise push a component above 1.0 and inflate the score.

WEIGHTS is a module constant with a comment. Every disagreement about a feed’s score is a disagreement about those three numbers, and having them in one visible place makes that argument short.

Components are stored, not just the score. DailyQuality carries eleven fields. The score makes feeds rankable; the rest make the ranking explicable, and a dashboard that shows only the score sends people back to the raw logs.

Verification and Output

python 
def verify(quality: DailyQuality) -> None:
    assert 0.0 <= quality.score <= 1.0, "score outside the unit interval"
    assert 0.0 <= quality.coverage_weighted <= 1.0
    assert 0.0 <= quality.match_rate_median <= 1.0
    assert quality.staleness_p50 <= quality.staleness_p95 <= quality.staleness_max, (
        "staleness percentiles are not ordered — check the percentile helper")
    assert quality.failed_polls <= quality.polls
    assert abs(sum(WEIGHTS.values()) - 1.0) < 1e-9, "component weights do not sum to 1"

The percentile ordering assertion is worth keeping. Off-by-one errors in percentile indexing are easy to introduce and produce numbers that look reasonable individually and are inconsistent with each other.

A week of one feed:

text 
INFO gtfs.rt.dashboard: mbta_vehicles 2026-08-01: score 0.99 (fresh 1.00, match 0.99, coverage 0.97), p95 staleness 41s, 0 min stale, 0 failed poll(s)
INFO gtfs.rt.dashboard: mbta_vehicles 2026-08-02: score 0.98 (fresh 1.00, match 0.98, coverage 0.95), p95 staleness 44s, 0 min stale, 0 failed poll(s)
INFO gtfs.rt.dashboard: mbta_vehicles 2026-08-03: score 0.71 (fresh 0.62, match 0.99, coverage 0.58), p95 staleness 1802s, 47 min stale, 12 failed poll(s)
INFO gtfs.rt.dashboard: mbta_vehicles 2026-08-04: score 0.99 (fresh 1.00, match 0.99, coverage 0.98), p95 staleness 39s, 0 min stale, 0 failed poll(s)

The third day tells the whole story in one line: freshness collapsed, matching was unaffected, coverage fell because a stale feed reports stale vehicles. That pattern — freshness and coverage down together, match rate flat — is the signature of a producer outage rather than a static-feed change, and it is legible without opening anything else.

Across agencies, the same records rank directly:

text 
feed              score  p95_staleness_s  match_median  coverage
metro_rail         0.99               38          0.99      0.98
city_bus           0.94              112          0.97      0.86
county_bus         0.81              340          0.91      0.62
regional_coach     0.55             2140          0.88      0.31
A week of one feed, read at a glance Daily quality across the three scored components: Monday's dip is a producer outage, visible because freshness and coverage fell together while matching did not. Mon Tue Wed Thu Fri Sat Sun Freshness Matching Coverage freshness and coverage falling together, matching flat, is a producer outage

Gotchas and Edge Cases

  • A feed with a genuinely partial fleet. An agency equipping half its vehicles will never exceed 0.5 coverage, and its score is permanently capped near 0.75. Either baseline coverage per feed against its own historical maximum, or report coverage separately and exclude it from the cross-agency score.
  • Days with very few polls. A monitoring restart can leave a service day with fifty samples. The percentiles are then meaningless; record the poll count in every row so a thin day is visible rather than being read as a real measurement.
  • Service days that are not 24 hours. A clock change makes one service day 23 or 25 hours long. Nothing here breaks, but minutes_stale and the hour buckets shift by one, which is worth knowing before someone investigates a phantom anomaly on two specific dates a year.
  • Storing per-poll data forever. A 15-second feed generates 5,760 rows a day, which across a hundred agencies is 200 million rows a year for data nobody queries after a fortnight. Keep the daily records permanently and expire the per-poll detail; the daily table is a few kilobytes per feed per year and belongs in the same partitioned store as the rest of the history.
  • Comparing scores after changing the weights. Every historical score becomes incomparable the moment WEIGHTS changes. Version the weights alongside the record if the score is ever used for a service-level commitment.

Frequently Asked Questions

Why percentiles rather than an average staleness?

Because staleness distributions have long tails. A feed that is perfect for 23 hours and frozen for one has an unremarkable mean and a terrible 95th percentile. The tail is the part riders experience, so the tail is what should be reported.

Should a single score replace the individual metrics?

No — it should sit alongside them. A score makes feeds comparable and trends visible; the components tell you what to do about it. A dashboard showing only a score sends people to the raw logs, which defeats the purpose.

Why weight coverage by scheduled service?

Because an unweighted daily average of hourly coverage gives the same weight to 03:00, when two trips are scheduled, as to 08:00, when four hundred are. That makes a feed that fails entirely during the morning peak look respectable.

How much history is worth keeping?

One row per feed per day indefinitely — it is a few kilobytes a year. Per-poll records are worth keeping for a couple of weeks for incident investigation, and are not worth keeping beyond that.