Alerting on Realtime Feed Outages

Alert on sustained degradation measured against what the timetable says should be running, not against zero. A feed reporting no vehicles at three in the morning on a network that closes at midnight is behaving correctly, and an alerting rule that cannot tell that apart from an outage will be muted within a week — after which real outages go unnoticed too. Require the same problem in several consecutive polls, escalate by how long it has persisted rather than by guessing a severity, and emit an explicit recovery event when it clears. The metrics being alerted on come from monitoring realtime feed quality.

Root Cause Analysis

Realtime alerting fails in two directions, and almost every team experiences both in sequence.

First, it is too loud. A naive rule — “alert if the feed reports fewer than N vehicles” — fires every night when service stops, every Sunday morning, and every public holiday. It also fires on every producer restart, every transient network fault and every garbage-collection pause. Within a fortnight the channel is muted.

Then it is too quiet. The muted channel means nobody sees the real outage. The feed freezes on a Friday evening, predictions stay frozen all weekend, and the first report comes from a rider on Monday.

The way out is to make the rule understand two things the naive version does not.

The first is the schedule. The question is never “are there vehicles” but “are there vehicles where the timetable says there should be”. That number comes from the static feed — the trips whose service is active today and whose span covers the current moment — and it is zero overnight, which makes the whole overnight problem disappear rather than needing to be suppressed with a time-of-day rule that then has to be maintained per agency and per holiday.

The second is persistence. A single degraded poll carries almost no information. The same degradation across five consecutive polls is a fact. Requiring persistence removes the transient noise without adding any delay that matters operationally.

There is a third element that is easy to skip and expensive to omit: recovery. Most realtime faults resolve themselves, often within minutes. An alerting system that fires and never closes leaves an engineer investigating a problem that has already ended, and it makes the fault history useless for judging whether a feed is actually reliable.

Production-Ready Python Implementation

"""Schedule-aware alerting for GTFS-Realtime feed health."""
from __future__ import annotations

import logging
import time
from collections import deque
from dataclasses import dataclass, field
from enum import IntEnum

log = logging.getLogger("gtfs.rt.alert")

SUSTAINED_POLLS = 5            # consecutive bad polls before anything fires
ESCALATE_AFTER_S = {600: "MAJOR", 3600: "CRITICAL"}
HISTORY = 200


class Severity(IntEnum):
    OK = 0
    MINOR = 1
    MAJOR = 2
    CRITICAL = 3


@dataclass(frozen=True)
class Health:
    """One poll's health, already judged by the individual metric monitors."""
    at: float
    problems: tuple[str, ...]
    scheduled_now: int          # trips the timetable says should be running
    entities: int

    @property
    def quiet_period(self) -> bool:
        """Nothing is scheduled, so nothing being reported is correct."""
        return self.scheduled_now == 0


@dataclass
class Alert:
    key: str                    # the problem text; stable across polls
    owner: str
    opened_at: float
    severity: Severity = Severity.MINOR
    closed_at: float | None = None

    @property
    def duration_s(self) -> float:
        return (self.closed_at or time.time()) - self.opened_at


# Who is actually able to fix each class of problem.
def owner_for(problem: str) -> str:
    lowered = problem.lower()
    if "stale" in lowered or "frozen" in lowered or "poll failed" in lowered:
        return "feed publisher"
    if "match rate" in lowered or "static feed" in lowered:
        return "static feed owner"
    if "coverage" in lowered:
        return "agency vehicle systems"
    return "on-call"


class AlertManager:
    def __init__(self, feed_name: str, sink=None):
        self.feed_name = feed_name
        self._history: deque[Health] = deque(maxlen=HISTORY)
        self._open: dict[str, Alert] = {}
        self._sink = sink or (lambda event, alert: log.warning(
            "%s %s [%s] %s (%.0fs)", event, alert.severity.name, alert.owner,
            alert.key, alert.duration_s))

    def observe(self, health: Health) -> None:
        self._history.append(health)

        # A quiet period cannot produce an outage, and must not close an open one
        # either — a feed that broke at 23:50 is still broken at 00:10.
        if health.quiet_period:
            log.debug("%s: quiet period, %d entities, nothing scheduled",
                      self.feed_name, health.entities)
            return

        recent = [h for h in list(self._history)[-SUSTAINED_POLLS:]
                  if not h.quiet_period]
        if len(recent) < SUSTAINED_POLLS:
            return

        sustained = set(recent[0].problems)
        for h in recent[1:]:
            sustained &= set(h.problems)

        now = health.at
        for problem in sorted(sustained):
            alert = self._open.get(problem)
            if alert is None:
                alert = Alert(key=problem, owner=owner_for(problem), opened_at=now)
                self._open[problem] = alert
                self._sink("OPENED", alert)
            else:
                escalated = self._severity_for(now - alert.opened_at)
                if escalated > alert.severity:
                    alert.severity = escalated
                    self._sink("ESCALATED", alert)

        for problem, alert in list(self._open.items()):
            if problem not in sustained and problem not in health.problems:
                alert.closed_at = now
                self._sink("RECOVERED", alert)
                del self._open[problem]

    @staticmethod
    def _severity_for(elapsed_s: float) -> Severity:
        level = Severity.MINOR
        for threshold, name in sorted(ESCALATE_AFTER_S.items()):
            if elapsed_s >= threshold:
                level = Severity[name]
        return level

    @property
    def open_alerts(self) -> list[Alert]:
        return sorted(self._open.values(), key=lambda a: a.opened_at)

Step-by-Step Walkthrough

quiet_period is derived from the schedule, not from the clock. scheduled_now comes from the static feed — trips whose service is active on today’s service date and whose first and last stop times bracket the current moment. That makes the rule correct on public holidays, during seasonal shutdowns and on a network that runs all night, none of which a time-of-day suppression window handles without per-agency maintenance.

A quiet period returns early without closing anything. This is the subtle half. If a feed breaks at 23:50 and service stops at midnight, the fault has not been fixed by the network closing — so quiet periods are skipped rather than treated as healthy. The alert stays open until a poll during actual service hours shows it resolved.

Sustained problems are the intersection across recent polls. A problem must be present in all five to fire. Using a count instead would let a feed alternating between two different faults never reach the threshold on either, while being continuously broken.

Quiet-period polls are filtered out of the persistence window. Otherwise a fault that begins just before service ends would need five polls during service hours and would have its window diluted by the overnight polls in between.

Severity comes from elapsed time, not from a judgement about the problem. Ten minutes is MAJOR, an hour is CRITICAL, regardless of which metric degraded. Guessing severity per problem type invites endless argument; duration is objective and correlates well with impact.

owner_for puts a team on every alert. The three failure classes have three different owners, and an alert that does not name one gets forwarded twice before it reaches someone who can act.

Recovery closes on either condition. A problem is closed when it is neither sustained nor present in the current poll, which means a fault that clears immediately still closes rather than waiting five more polls for the window to empty.

Verification and Output

def verify(manager: AlertManager) -> None:
    for alert in manager.open_alerts:
        assert alert.closed_at is None, "an open alert carries a close time"
        assert alert.duration_s >= 0, "alert duration runs backwards"
        assert alert.owner, "alert has no owner"

    keys = [a.key for a in manager.open_alerts]
    assert len(keys) == len(set(keys)), "the same problem is open twice"

    history = list(manager._history)
    stamps = [h.at for h in history]
    assert stamps == sorted(stamps), "health observations recorded out of order"
    assert all(h.scheduled_now >= 0 for h in history), "negative scheduled count"

A night that should not page anyone:

DEBUG gtfs.rt.alert: county_bus: quiet period, 0 entities, nothing scheduled
DEBUG gtfs.rt.alert: county_bus: quiet period, 0 entities, nothing scheduled

A real outage, escalating and then recovering:

WARN gtfs.rt.alert: OPENED MINOR [feed publisher] stale: snapshot is 214s old, above the 90s threshold (30s cadence x 3) (0s)
WARN gtfs.rt.alert: ESCALATED MAJOR [feed publisher] stale: snapshot is 786s old, above the 90s threshold (30s cadence x 3) (612s)
WARN gtfs.rt.alert: RECOVERED MAJOR [feed publisher] stale: snapshot is 786s old, above the 90s threshold (30s cadence x 3) (994s)

Sixteen minutes, opened and closed automatically, with the publisher named. That record is also what any monthly reliability figure should be built from — counting opened alerts and summing their durations is a far better measure of a feed than an uptime percentage taken from HTTP checks.

Gotchas and Edge Cases

• The problem text is the alert key. If a message embeds a changing number — “snapshot is 214s old” — every poll produces a new key and every poll opens a new alert. Either keep the volatile detail out of the key or normalise it before using it; the implementation above deliberately opens on the first message and keeps it, which works because the intersection only matches identical strings and therefore requires the text to be stable across polls. Where a metric monitor produces varying text, template it.
• Feeds with no static counterpart. Some realtime endpoints cover a network whose static feed you do not have. scheduled_now is then unknowable, and the quiet-period logic cannot work. Fall back to a learned baseline of entity counts by hour of week rather than assuming service is always expected.
• Very long quiet periods. A seasonal operator may be quiet for months. The persistence window will never fill, and no alert can ever fire — which is correct, but it means a fault introduced during the off season is only discovered on the first day of service. Run a lighter check that simply confirms the endpoint still answers.
• Escalation thresholds during known maintenance. Agencies do announce realtime maintenance windows. A suppression list keyed on feed and time window is worth having, but it should suppress paging, not recording, or the reliability history quietly excludes every known outage.
• Clock source for health.at. Use the same clock throughout. Mixing the poll’s fetch time with the alert manager’s own time.time() produces durations that drift, which is why Alert.duration_s falls back to time.time() only when the alert is still open.

Frequently Asked Questions

Related

• Measuring GTFS-Realtime Feed Staleness — where the staleness problems being alerted on come from
• Building a Realtime Feed Quality Dashboard — the view that makes an alert interpretable
• Up: Monitoring Realtime Feed Quality — the metrics behind every alert here
• Section: GTFS-Realtime Integration · Home