Detecting Service Gaps and Feed Expiry

Expand the calendar into the set of dates that carry service, then measure two things: the days inside the window where no service_id at all is active, and the number of days between today and the last active date. The first finds the holes; the second finds the cliff. A feed with fewer than 30 days of remaining coverage is on its way to failing, and one with fewer than 7 is already an incident. Both numbers fall straight out of the service map built in calendar and service exception modeling, and neither is reported by a schema validator.

Root Cause Analysis

A GTFS feed is a snapshot with an expiry date, but nothing in the format makes the expiry prominent. feed_info.txt may carry feed_start_date and feed_end_date, and those fields are optional and frequently wrong — they describe what the publisher intended, not what the calendar files actually contain. The real horizon is the largest date any service_id is active on, and it has to be computed.

Two distinct failures hide behind that computation.

Expiry is the simple one. An agency republishes every fortnight; one publication is missed; three weeks later every query returns no trips. Applications built on the feed usually degrade badly here, because “no trips today” and “no service today” are indistinguishable downstream. The rider sees an empty departure board on a line that is running normally.

Gaps are subtler. A gap is a date inside the feed’s own window where no service is active at all. Sometimes this is real — an operator that genuinely runs nothing on a public holiday, or a seasonal service outside its season. More often it is an artefact: the agency added exception_type = 2 removals for a week of engineering work and never added the replacement pattern, so the feed asserts that the entire network stops for a week. Nothing about that is invalid GTFS. Every schema check passes. The only way to catch it is to look at the shape of the coverage.

The two measurements share a foundation, so they belong in the same pass over the same expanded calendar.

Production-Ready Python Implementation

"""Measure a GTFS feed's service coverage: interior gaps and remaining horizon."""
from __future__ import annotations

import logging
from dataclasses import dataclass, field
from datetime import date, datetime, timedelta
from pathlib import Path
from zipfile import ZipFile, BadZipFile
from zoneinfo import ZoneInfo

import pandas as pd

log = logging.getLogger("gtfs.calendar.coverage")

HORIZON_WARN_DAYS = 30
HORIZON_CRITICAL_DAYS = 7
GAP_REPORT_LIMIT = 10


@dataclass
class Coverage:
    first_date: date
    last_date: date
    served_days: int
    gaps: list[tuple[date, date]] = field(default_factory=list)
    timezone: str = "UTC"

    def days_remaining(self, today: date | None = None) -> int:
        today = today or datetime.now(ZoneInfo(self.timezone)).date()
        return (self.last_date - today).days

    def severity(self, today: date | None = None) -> str:
        left = self.days_remaining(today)
        if left < 0:
            return "EXPIRED"
        if left < HORIZON_CRITICAL_DAYS:
            return "CRITICAL"
        if left < HORIZON_WARN_DAYS:
            return "WARNING"
        return "OK"


def _agency_timezone(archive: ZipFile) -> str:
    """The feed's own timezone — the horizon is a local-date question."""
    try:
        with archive.open("agency.txt") as fh:
            agency = pd.read_csv(fh, dtype="string")
    except KeyError:
        return "UTC"
    zones = [z for z in agency.get("agency_timezone", pd.Series(dtype="string")).dropna()]
    if not zones:
        return "UTC"
    if len(set(zones)) > 1:
        log.warning("feed spans %d timezones; using %s for the horizon",
                    len(set(zones)), zones[0])
    return zones[0]


def measure_coverage(feed_path: Path, active: pd.DataFrame) -> Coverage:
    """active: the expanded (service_id, date) map for this feed."""
    if active.empty:
        raise ValueError(f"{feed_path.name}: no active service dates to measure")

    served = sorted({d.date() if hasattr(d, "date") else d for d in active["date"]})
    first, last = served[0], served[-1]

    # A gap is a run of consecutive days inside the window with no service at all.
    gaps: list[tuple[date, date]] = []
    for previous, following in zip(served, served[1:]):
        if (following - previous).days > 1:
            gaps.append((previous + timedelta(days=1), following - timedelta(days=1)))

    try:
        with ZipFile(feed_path) as archive:
            tz = _agency_timezone(archive)
    except (BadZipFile, OSError):
        tz = "UTC"

    coverage = Coverage(first_date=first, last_date=last, served_days=len(served),
                        gaps=gaps, timezone=tz)

    left = coverage.days_remaining()
    log.info("%s: %s..%s, %d served day(s), %d gap(s), %d day(s) of coverage left [%s]",
             feed_path.name, first, last, len(served), len(gaps), left,
             coverage.severity())
    for start, end in gaps[:GAP_REPORT_LIMIT]:
        span = (end - start).days + 1
        log.warning("no service at all from %s to %s (%d day%s)",
                    start, end, span, "" if span == 1 else "s")
    if len(gaps) > GAP_REPORT_LIMIT:
        log.warning("...and %d further gap(s)", len(gaps) - GAP_REPORT_LIMIT)
    return coverage


def enforce(coverage: Coverage) -> None:
    """Refuse to serve a feed that no longer describes current service."""
    level = coverage.severity()
    if level == "EXPIRED":
        raise RuntimeError(
            f"feed expired on {coverage.last_date}; serving it would invent schedule")
    if level == "CRITICAL":
        log.error("only %d day(s) of coverage remain — page the feed owner now",
                  coverage.days_remaining())

Step-by-Step Walkthrough

served is a set of dates, deduplicated across services. The question a gap answers is “does anything run that day”, so the per-service detail is collapsed first. A day on which only one obscure service_id is active is still a served day, and rightly so — the network is not shut.

Gaps come from consecutive pairs, not from a full date range. Zipping the sorted list against itself offset by one finds every discontinuity in a single pass, and it naturally reports each gap as an interval rather than as a list of individual missing days. An engineering closure then shows up as one line in the log instead of fourteen.

The horizon is computed in the agency’s timezone. _agency_timezone reads agency_timezone from the feed itself, because “how many days are left” is a question about the local date. Using a UTC date can be a day out in either direction, and that is exactly the day it matters. Where a feed spans several zones the first is used and the fact is logged — a multi-timezone feed needs its own handling throughout.

severity is separate from measure_coverage. Measuring is a pure function of the feed; deciding what to do about the result is policy, and policy changes. Keeping them apart means the thresholds can be tuned per agency without touching the measurement.

enforce raises on an expired feed rather than logging. This is deliberate and it is the whole point of the module. A pipeline that logs an expiry and carries on will serve the last known schedule indefinitely, which is worse than serving nothing: it tells riders a bus is coming when the feed has no idea whether it is.

GAP_REPORT_LIMIT bounds the output, and says so. A seasonal feed can have hundreds of gaps, and a log that prints all of them is a log nobody reads. The count of what was suppressed is printed too, so the truncation is never mistaken for completeness.

Verification and Output

def verify_coverage(coverage: Coverage) -> None:
    assert coverage.first_date <= coverage.last_date, "window runs backwards"
    span = (coverage.last_date - coverage.first_date).days + 1
    gap_days = sum((e - s).days + 1 for s, e in coverage.gaps)
    assert coverage.served_days + gap_days == span, (
        f"served {coverage.served_days} + gaps {gap_days} != window {span}")
    assert all(s <= e for s, e in coverage.gaps), "a gap runs backwards"

The identity in the middle is the one that matters: served days plus gap days must exactly equal the window. If it does not, the gap detection has missed something, and it will have missed it silently.

A healthy feed:

INFO  gtfs.calendar.coverage: mbta_20261019.zip: 2026-10-19..2027-03-13, 146 served day(s), 0 gap(s), 218 day(s) of coverage left [OK]

A feed with the engineering-closure artefact:

INFO  gtfs.calendar.coverage: regional_20261102.zip: 2026-11-02..2026-12-27, 49 served day(s), 1 gap(s), 51 day(s) of coverage left [OK]
WARN  gtfs.calendar.coverage: no service at all from 2026-11-23 to 2026-11-29 (7 days)

Coverage looks fine and the schema is valid, but the feed asserts that the entire network stops for a week in November. That is the defect this check exists to surface, and it needs a human to decide whether the shutdown is real.

Gotchas and Edge Cases

• feed_info.txt disagreeing with the calendar. Common, and the calendar wins. feed_end_date is a statement of intent; the last active service date is what the feed can actually answer questions about. Report the disagreement, then use the calendar.
• Christmas Day and similar single-day gaps. Real for some operators. Distinguish them by whether the same date is a gap in previous publications too — a recurring annual gap is policy, a novel one is a mistake.
• Seasonal operators. A ferry that runs May to September produces an enormous gap that is entirely correct. These feeds need per-agency thresholds rather than a global rule, which is why severity is kept separate from the measurement.
• A feed republished with a shorter window than the one it replaces. Nothing here catches that on its own, because each measurement looks at one feed. Comparing the horizon against the previous publication is the check that finds it, and it belongs with feed version control.
• Timezone data missing from the container. ZoneInfo raises if the system has no tz database and tzdata is not installed. Pin tzdata in any slim container image, or the horizon check fails in production and nowhere else.

Frequently Asked Questions

Related

• Expanding GTFS Calendar to Service Dates in Python — where the service map this measures comes from
• Modeling Holiday Service in GTFS — the exceptions that turn into gaps when the replacement pattern is forgotten
• Up: Calendar and Service Exception Modeling — the calendar model behind both measurements
• Section: GTFS Feed Architecture & Fundamentals · Home