1. Home
  2. GTFS Realtime Integration
  3. Monitoring Realtime Feed Quality
  4. Measuring GTFS Realtime Feed Staleness

Measuring GTFS-Realtime Feed Staleness

Staleness is now - header.timestamp, using the FeedMessage header clock and never the HTTP response headers — a CDN will happily serve a forty-minute-old payload with a fresh Date. Set the alert threshold at about three times the publisher’s observed cadence, derived from the distinct header timestamps you have seen, rather than from a hard-coded number that suits one feed and not the next. Then watch for negative staleness, which means clock skew and invalidates every figure until it is fixed. This is the first of the four metrics in monitoring realtime feed quality.

Four clocks, one of which is the right one A grid over the timestamps available on a poll and what each actually describes. Describes Use for staleness HTTP Date when the response was assembled no Last-Modified the cache entry no Your fetch time your own scheduler only as the other operand header.timestamp when the producer built it yes

Root Cause Analysis

The failure this metric exists to catch is a feed that keeps working and stops updating. The endpoint returns 200, the body is valid protobuf, the entity list is fully populated — and every vehicle in it is where it was half an hour ago. Nothing about the request failed, so uptime monitoring reports the feed as healthy while every prediction built on it is fiction.

Three things make this harder than a subtraction.

The obvious clocks are the wrong clocks. The HTTP Date header describes when the response was assembled, which for a cached payload is unrelated to when the data was generated. Last-Modified, where present, describes the cache entry. The moment you fetched it describes your own scheduler. Only header.timestamp — set by the producer when it built the message — describes the data.

The threshold is feed-specific. Agencies publish at wildly different rates, from every five seconds for a busy metro to every five minutes for a regional bus network. Ninety seconds of staleness is an incident on the first and entirely normal on the second. A hard-coded threshold is therefore either too noisy or too quiet, and usually both across a portfolio of feeds.

Clock skew is silent. If the monitoring host’s clock drifts thirty seconds fast, every staleness figure is thirty seconds too high and still looks plausible. The only tell is that a feed occasionally reports a negative age, which is impossible and points straight at the cause — and which code that clamps to zero will hide forever.

There is a fourth, subtler case: a fresh header over stale entities. The producer is running, generating a new message every fifteen seconds, and the vehicle positions inside it have not changed because the upstream AVL system is down. The header clock says everything is fine. Per-entity timestamps are the only thing that says otherwise.

Production-Ready Python Implementation

python 
"""Staleness monitoring for a GTFS-Realtime feed."""
from __future__ import annotations

import logging
import statistics
import time
from collections import deque
from dataclasses import dataclass

from google.transit import gtfs_realtime_pb2

log = logging.getLogger("gtfs.rt.staleness")

STALENESS_MULTIPLIER = 3.0        # alert above this many publication intervals
FALLBACK_CADENCE_S = 60.0         # used until enough distinct timestamps are seen
MIN_SAMPLES_FOR_CADENCE = 4
CLOCK_SKEW_TOLERANCE_S = 5.0      # allow this much before calling it skew
HISTORY = 400


@dataclass(frozen=True)
class Snapshot:
    fetched_at: float
    header_timestamp: int
    entity_timestamps: tuple[int, ...]

    @property
    def age_s(self) -> float:
        return self.fetched_at - self.header_timestamp

    @property
    def median_entity_age_s(self) -> float | None:
        if not self.entity_timestamps:
            return None
        return self.fetched_at - statistics.median(self.entity_timestamps)


class StalenessMonitor:
    def __init__(self, feed_name: str):
        self.feed_name = feed_name
        self._history: deque[Snapshot] = deque(maxlen=HISTORY)
        self._seen_headers: deque[int] = deque(maxlen=HISTORY)

    def observe(self, payload: bytes, fetched_at: float | None = None) -> Snapshot:
        fetched_at = time.time() if fetched_at is None else fetched_at
        message = gtfs_realtime_pb2.FeedMessage()
        message.ParseFromString(payload)

        header_ts = int(message.header.timestamp)
        stamps = []
        for entity in message.entity:
            if entity.HasField("vehicle") and entity.vehicle.timestamp:
                stamps.append(int(entity.vehicle.timestamp))
            elif entity.HasField("trip_update") and entity.trip_update.timestamp:
                stamps.append(int(entity.trip_update.timestamp))

        if header_ts == 0:
            # Required by the spec; fall back to the entities so we report SOMETHING
            # rather than an age of "now", which would look perfectly fresh.
            if stamps:
                header_ts = max(stamps)
                log.warning("%s: header.timestamp is zero — using the newest entity "
                            "timestamp instead", self.feed_name)
            else:
                log.error("%s: no timestamp anywhere in the message; staleness is "
                          "unmeasurable for this poll", self.feed_name)

        snapshot = Snapshot(fetched_at=fetched_at, header_timestamp=header_ts,
                            entity_timestamps=tuple(stamps))
        self._history.append(snapshot)
        if header_ts and (not self._seen_headers or self._seen_headers[-1] != header_ts):
            self._seen_headers.append(header_ts)
        return snapshot

    def cadence_s(self) -> float | None:
        """The producer's real publication interval, from DISTINCT header stamps."""
        stamps = sorted(set(self._seen_headers))
        if len(stamps) < MIN_SAMPLES_FOR_CADENCE:
            return None
        gaps = [b - a for a, b in zip(stamps, stamps[1:]) if b > a]
        return statistics.median(gaps) if gaps else None

    def threshold_s(self) -> float:
        return (self.cadence_s() or FALLBACK_CADENCE_S) * STALENESS_MULTIPLIER

    def assess(self, snapshot: Snapshot) -> list[str]:
        problems = []

        if snapshot.age_s < -CLOCK_SKEW_TOLERANCE_S:
            problems.append(
                f"clock skew: header is {-snapshot.age_s:.0f}s in the FUTURE — every "
                "staleness figure from this host is wrong until the clocks agree")
            return problems      # nothing else measured here can be trusted

        limit = self.threshold_s()
        if snapshot.age_s > limit:
            problems.append(
                f"stale: snapshot is {snapshot.age_s:.0f}s old, above the "
                f"{limit:.0f}s threshold ({self.cadence_s() or FALLBACK_CADENCE_S:.0f}s "
                f"cadence x {STALENESS_MULTIPLIER:g})")

        entity_age = snapshot.median_entity_age_s
        if entity_age is not None and entity_age > limit * 2:
            problems.append(
                f"fresh header over stale entities: median entity age "
                f"{entity_age:.0f}s against a header age of {snapshot.age_s:.0f}s — "
                "the producer is running but its data source may not be")
        return problems

    def frozen_for_s(self) -> float:
        """How long the header timestamp has not advanced, regardless of polling."""
        if len(self._history) < 2:
            return 0.0
        newest = self._history[-1]
        for snapshot in reversed(self._history):
            if snapshot.header_timestamp != newest.header_timestamp:
                return newest.fetched_at - snapshot.fetched_at
        return newest.fetched_at - self._history[0].fetched_at
A feed that keeps answering and stops updating The header timestamp stops advancing while polls continue to succeed, so uptime monitoring sees nothing at all. 0 min 5 min 10 min 15 min 20 min last real update header stops advancing still returning 200 and still 15 minutes stale alert at about three publication intervals, derived rather than configured

Step-by-Step Walkthrough

observe takes bytes, not a response object. Staleness is a property of the message, so the monitor never sees the HTTP layer and cannot be tempted to read a Date header. That separation is the point: everything it measures comes from what the producer actually said.

A zero header timestamp falls back to the newest entity timestamp, loudly. The field is required, but feeds omit it. Reporting an age of zero in that case would be the worst outcome, because a broken feed would look perfectly fresh. Using the newest entity stamp gives a lower bound on the age; logging at warning level makes clear the number is a substitute.

_seen_headers only records a value when it changes. Polling every 10 seconds against a producer publishing every 30 returns the same message three times. Appending only on change means the deque holds distinct publications, and the median gap between them is the real cadence rather than the polling interval.

Clock skew short-circuits everything. When the header is meaningfully in the future, assess returns immediately. Continuing would produce a staleness number, and that number would be wrong by exactly the skew — the kind of plausible-looking figure that gets trusted for months.

Entity age is compared against twice the header threshold. Individual entities are legitimately older than the message that carries them: a vehicle reporting every 30 seconds appears in a message generated 5 seconds ago with a timestamp 25 seconds old. Doubling the allowance leaves room for that while still catching a fleet that has stopped reporting entirely.

frozen_for_s measures the feed, not the poller. It walks back through history to the last time the header changed and returns the elapsed wall-clock time. That is the number to put in an alert, because “the feed has not updated in 22 minutes” is actionable in a way that “staleness 1,340 seconds” is not.

Verification and Output

python 
def verify(monitor: StalenessMonitor) -> None:
    history = list(monitor._history)
    fetches = [s.fetched_at for s in history]
    assert fetches == sorted(fetches), "polls recorded out of order"

    headers = [s.header_timestamp for s in history if s.header_timestamp]
    assert headers == sorted(headers), (
        "header timestamps went backwards — the producer may be load-balanced "
        "across nodes with different state")

    cadence = monitor.cadence_s()
    if cadence is not None:
        assert cadence > 0, "derived a non-positive publication cadence"
        assert monitor.threshold_s() >= cadence, "threshold below one publication interval"

Normal operation on a 30-second feed:

text 
INFO gtfs.rt.staleness: mbta_vehicles: age 12s, cadence 30s, threshold 90s, 412 entities, median entity age 26s

The failure this page exists for:

text 
WARN gtfs.rt.staleness: mbta_vehicles: stale: snapshot is 1347s old, above the 90s threshold (30s cadence x 3)
WARN gtfs.rt.staleness: mbta_vehicles: frozen for 1338s — the endpoint is answering but the data has not changed

And the one that only per-entity timestamps reveal:

text 
WARN gtfs.rt.staleness: county_bus: fresh header over stale entities: median entity age 903s against a header age of 8s — the producer is running but its data source may not be
Three staleness readings and what each means The sign and shape of the measurement identifies the fault before anyone opens a log. What does the staleness figure look like? large and growing Frozen feed the producer stopped; page the publisher negative Clock skew fix the host clock; every figure is wrong small, entities old Dead data source the producer is fine, the AVL is not

Gotchas and Edge Cases

  • Producers that round their timestamp. Some emit whole minutes, which makes the derived cadence granular and can push a genuinely 30-second feed to a 60-second reading. The threshold multiplier absorbs it; a tighter multiplier would not.
  • A restarted monitor. cadence_s returns None until four distinct timestamps are seen, so the first minute or so uses FALLBACK_CADENCE_S. That is deliberate — better a rough threshold than an alert storm from a single-sample estimate.
  • Feeds behind an aggressive cache. The header timestamp will step in whole cache-TTL increments, which reads as a slower cadence than the producer’s own. That is the correct thing to measure: it is the age of the data you can see, which is what your predictions are built on.
  • Daylight-saving transitions. POSIX timestamps are unaffected, which is one of the few places in GTFS work where the DST problem does not arise. Do not convert to local time to compute an age.
  • Containers without NTP. The most common source of real skew. The clock-skew branch will fire intermittently as drift crosses the tolerance, which reads as a flapping alert until someone looks at the host clock.

Frequently Asked Questions

Why not use the HTTP Date or Last-Modified header?

Because they describe the response, not the data. A CDN can serve a forty-minute-old payload with a Date header from one second ago, and a producer can regenerate an identical message without any of its content being newer. Only header.timestamp says when the producer built the snapshot.

What if header.timestamp is zero or missing?

It is required, so treat its absence as a feed defect and fall back to per-entity timestamps if any exist. If nothing carries a timestamp, staleness cannot be measured at all — record that fact explicitly rather than reporting zero, which would look like a perfectly fresh feed.

Why deduplicate timestamps before deriving the cadence?

Because polling faster than the producer publishes returns the same message repeatedly. Counting those repeats as observations yields a median gap of zero and a staleness threshold of zero, which then flags every poll.

Can staleness be negative?

Only through clock skew. A header timestamp ahead of your own clock means one of the two machines is wrong, and every staleness figure computed against it is wrong by the same amount. Detect it and fix the clock rather than clamping the number to zero.