1. Home
  2. GTFS Feed Architecture Fundamentals
  3. Calendar And Service Exception Modeling
  4. Expanding GTFS Calendar To Service Dates In Python

Expanding GTFS Calendar to Service Dates in Python

Expand calendar.txt by cross-joining each weekday flag against the feed’s full date range and clipping to each pattern’s start_date and end_date, then concatenate the calendar_dates.txt rows with exception_type = 1 and subtract those with exception_type = 2. The order is not negotiable: exceptions are applied after expansion, because an added date is allowed to fall outside the pattern’s window. The result is one row per active (service_id, date) pair — the index every other calendar question resolves against, and the foundation of the wider calendar and service exception model.

Root Cause Analysis

The awkwardness here is that GTFS stores service in two incompatible shapes. calendar.txt is a rule: seven booleans and a date range, from which the actual service dates have to be derived. calendar_dates.txt is a fact: this service, this date, added or removed. Nothing downstream can consume a rule — trips.txt joins on service_id, and answering “does this trip run on 14 March” means knowing whether that service_id is live on that date. So the rule has to be materialised into facts before anything else happens.

Two mistakes account for nearly every calendar bug seen in production. The first is expanding the pattern and stopping there, never reading calendar_dates.txt at all. That feed will confidently report normal weekday service on Christmas Day, because the weekday flag for Thursday is set and the removal row was never applied. The second is applying the exceptions in the wrong order — filtering the combined set back through the start_date/end_date window, which silently discards every exception_type = 1 row an agency added for a date beyond the pattern’s end. Both defects are invisible in testing, because both produce a schedule that looks entirely plausible.

There is a third, subtler trap: comparing YYYYMMDD values as strings or integers rather than parsing them. It works until a feed pads a column differently, or a column picks up a stray space, or someone adds one to a date and produces 20251032. The rule is the same one that governs time fields elsewhere in the feed: parse once at the boundary, then work in the real type.

The only order that respects the specification Expand the weekly rule first, then add exception_type 1 dates, then subtract exception_type 2 dates — reordering any pair changes the answer. Expand weekly flags → dates Add exception_type 1 Subtract exception_type 2 Index by date an added date may legally fall outside the pattern's own window

Production-Ready Python Implementation

python 
"""Expand a GTFS feed's service patterns into a date-indexed service map."""
from __future__ import annotations

import logging
from datetime import date
from pathlib import Path
from zipfile import ZipFile

import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")
log = logging.getLogger("gtfs.calendar.expand")

DAY_COLUMNS = ("monday", "tuesday", "wednesday", "thursday",
               "friday", "saturday", "sunday")

CALENDAR_DTYPES = {
    "service_id": "string", "start_date": "string", "end_date": "string",
    **{day: "int8" for day in DAY_COLUMNS},
}
EXCEPTION_DTYPES = {
    "service_id": "string", "date": "string", "exception_type": "int8",
}


def _read(archive: ZipFile, member: str, dtypes: dict) -> pd.DataFrame:
    if member not in archive.namelist():
        log.info("%s absent — treating it as empty", member)
        return pd.DataFrame({c: pd.Series(dtype=d) for c, d in dtypes.items()})
    with archive.open(member) as fh:
        frame = pd.read_csv(fh, dtype=dtypes, keep_default_na=False, na_values=[""])
    log.info("%s: %d row(s)", member, len(frame))
    return frame


def _dates(series: pd.Series) -> pd.Series:
    parsed = pd.to_datetime(series, format="%Y%m%d", errors="coerce")
    bad = int((parsed.isna() & series.notna()).sum())
    if bad:
        log.error("%d unparseable YYYYMMDD value(s) — those rows are dropped", bad)
    return parsed


def expand_service_dates(feed_path: Path) -> pd.DataFrame:
    """Return one row per active (service_id, date), exceptions applied."""
    with ZipFile(feed_path) as archive:
        calendar = _read(archive, "calendar.txt", CALENDAR_DTYPES)
        exceptions = _read(archive, "calendar_dates.txt", EXCEPTION_DTYPES)

    if calendar.empty and exceptions.empty:
        raise ValueError(f"{feed_path.name} defines no service at all")

    frames: list[pd.DataFrame] = []

    if not calendar.empty:
        starts, ends = _dates(calendar["start_date"]), _dates(calendar["end_date"])
        valid = starts.notna() & ends.notna() & (ends >= starts)
        if not valid.all():
            for sid in calendar.loc[~valid, "service_id"]:
                log.error("service %s has an unusable date range and is skipped", sid)
        patterns = calendar[valid].reset_index(drop=True)
        starts, ends = starts[valid].reset_index(drop=True), ends[valid].reset_index(drop=True)

        span = pd.date_range(starts.min(), ends.max(), freq="D")
        bounds = pd.DataFrame({"service_id": patterns["service_id"],
                               "start": starts, "end": ends})

        # One merge per weekday: every pattern flagged for that weekday is paired with
        # every date that falls on it, then clipped to the pattern's own window.
        for weekday, column in enumerate(DAY_COLUMNS):
            days = span[span.weekday == weekday]
            active = patterns.loc[patterns[column] == 1, ["service_id"]]
            if days.empty or active.empty:
                continue
            paired = active.merge(pd.DataFrame({"date": days}), how="cross")
            paired = paired.merge(bounds, on="service_id")
            inside = (paired["date"] >= paired["start"]) & (paired["date"] <= paired["end"])
            frames.append(paired.loc[inside, ["service_id", "date"]])

    expanded = (pd.concat(frames, ignore_index=True) if frames
                else pd.DataFrame({"service_id": pd.Series(dtype="string"),
                                   "date": pd.Series(dtype="datetime64[ns]")}))

    if not exceptions.empty:
        ex = exceptions.assign(date=_dates(exceptions["date"]))
        ex = ex[ex["date"].notna() & ex["exception_type"].isin([1, 2])]
        added = ex.loc[ex["exception_type"] == 1, ["service_id", "date"]]
        removed = ex.loc[ex["exception_type"] == 2, ["service_id", "date"]]

        # Additions first — an added date may legitimately sit outside the pattern window.
        expanded = pd.concat([expanded, added], ignore_index=True)
        expanded = expanded.drop_duplicates(["service_id", "date"])

        if len(removed):
            drop = set(zip(removed["service_id"], removed["date"]))
            keep = [(s, d) not in drop
                    for s, d in zip(expanded["service_id"], expanded["date"])]
            expanded = expanded[keep]

    expanded = (expanded.drop_duplicates(["service_id", "date"])
                        .sort_values(["date", "service_id"])
                        .reset_index(drop=True))
    log.info("expanded to %d active (service_id, date) pair(s) across %s..%s",
             len(expanded), expanded["date"].min().date(), expanded["date"].max().date())
    return expanded


def services_on(expanded: pd.DataFrame, day: date) -> set[str]:
    """The service_id values active on one calendar date."""
    stamp = pd.Timestamp(day)
    return set(expanded.loc[expanded["date"] == stamp, "service_id"])


if __name__ == "__main__":
    active = expand_service_dates(Path("feeds/current.zip"))
    print(sorted(services_on(active, date(2026, 12, 25))))

Step-by-Step Walkthrough

The seven merges. The loop over DAY_COLUMNS is the heart of the expansion. For weekday index 0 (Monday) it selects every pattern whose monday flag is 1, pairs it with every Monday in the feed’s overall span, and keeps the pairs that fall inside that pattern’s own window. Seven passes cover the week. This replaces a day-by-day loop over each pattern’s range with seven vectorised merges, and it is what makes the function usable on a feed with tens of thousands of service patterns.

span is the union, not the per-pattern range. pd.date_range(starts.min(), ends.max()) covers the whole feed. Individual patterns are clipped afterwards by the inside mask. Building a separate range per pattern would put the loop straight back in.

how="cross" needs pandas 1.2 or later. On older versions the idiom is to add a constant key column to both frames and merge on it, which is what the equivalent code in the topic overview does. The cross join is clearer where it is available.

Additions are concatenated, removals are subtracted. The two operations are deliberately asymmetric. Additions go through concat and then drop_duplicates, so adding a date that the pattern already produced is a no-op rather than a duplicate row. Removals are applied last through a set membership test, which means a date that is both added and removed ends up removed.

The removal set is a Python set of tuples, not a merge. For the row counts involved — removals are rarely more than a few hundred rows — a set lookup is faster than an anti-join and far easier to read. If a feed ever produces removals in the hundreds of thousands, swap it for a merge(..., how="left", indicator=True) and filter on left_only.

Dates stay as pandas Timestamp, not Python date. Mixing the two in the same column is what produces comparisons that silently return False. services_on converts its argument rather than converting the column.

Expansion time by implementation, 30,000 service patterns Wall-clock seconds to expand a pathological feed that emits one service pattern per trip: the seven-merge form is what makes that feed tractable at all. Day-by-day Python loop 90 s one iteration per pattern per day Seven vectorised merges 1.4 s one merge per weekday Cached from a previous run 0.02 s keyed on the feed checksum on an ordinary feed of 40 patterns both forms finish instantly

Verification and Output

python 
def verify_expansion(expanded: pd.DataFrame, feed_path: Path) -> None:
    with ZipFile(feed_path) as archive:
        calendar = _read(archive, "calendar.txt", CALENDAR_DTYPES)
        exceptions = _read(archive, "calendar_dates.txt", EXCEPTION_DTYPES)
        with archive.open("trips.txt") as fh:
            trips = pd.read_csv(fh, dtype="string", usecols=["trip_id", "service_id"])

    assert not expanded.duplicated(["service_id", "date"]).any(), "duplicate service dates"

    # Every exception_type 1 date really is present.
    added = exceptions[exceptions["exception_type"] == 1]
    want = set(zip(added["service_id"], _dates(added["date"])))
    have = set(zip(expanded["service_id"], expanded["date"]))
    assert want <= have, f"{len(want - have)} added date(s) missing from the expansion"

    # Every exception_type 2 date really is gone.
    removed = exceptions[exceptions["exception_type"] == 2]
    gone = set(zip(removed["service_id"], _dates(removed["date"])))
    assert not (gone & have), f"{len(gone & have)} removed date(s) still active"

    # Every service_id a trip points at resolves to at least one date.
    referenced = set(trips["service_id"])
    resolvable = set(expanded["service_id"])
    stranded = referenced - resolvable
    if stranded:
        n = int(trips["service_id"].isin(stranded).sum())
        log.error("%d trip(s) reference %d service pattern(s) that never run",
                  n, len(stranded))

Run against a healthy metropolitan feed the output is unremarkable, which is the point:

text 
INFO gtfs.calendar.expand: calendar.txt: 24 row(s)
INFO gtfs.calendar.expand: calendar_dates.txt: 186 row(s)
INFO gtfs.calendar.expand: expanded to 8104 active (service_id, date) pair(s) across 2026-09-01..2026-12-19

The number worth watching is the date span. A feed whose last active date is inside the next fortnight is about to stop describing service altogether, which is the check covered in detecting service gaps and feed expiry.

The key the whole expansion produces Every downstream calendar question is a lookup on the pair of a service identifier and a concrete date. service_id + date The value trips.txt references; one trip has exactly one A concrete calendar date, never a weekday name or an offset build the pair once per feed version; every later question is a set lookup

Gotchas and Edge Cases

  • A feed with no calendar.txt at all. The function handles it — frames stays empty and the exception layer supplies every date — but the assertion in verify_expansion that every added date is present becomes the only real check you have. These feeds deserve their own handling.
  • One service_id per trip. Some scheduling systems emit a distinct service pattern for every trip, turning 24 rows into 30,000. The seven-merge form absorbs this; a day-by-day loop does not. Watch the memory too: the expansion is a cross product, and a two-year window over 30,000 patterns is millions of rows before the clip.
  • start_date equal to end_date. Perfectly valid — a single-day pattern. It only produces a row if that day’s weekday flag is set, which catches out agencies who set the range and forget the flags.
  • Timezone-aware dates. Do not localise the service date. A GTFS service date is a calendar label, not an instant; attaching a timezone to it invites a conversion that shifts it by a day. The instant only enters when a clock time is added to the date.

Frequently Asked Questions

Should exceptions be applied before or after expanding the weekly pattern?

After, always. calendar_dates.txt is an exception layer on top of the expanded pattern. Applying it first would let an exception_type = 1 date be filtered back out by the pattern’s start_date/end_date bounds, which the specification does not permit — an added date may fall outside the window entirely.

Is the vectorised expansion worth the extra complexity?

Only above roughly a thousand service patterns. Below that a plain day-by-day loop finishes in well under a second and is easier to trust. The vectorised version becomes essential for feeds that emit one service_id per trip and for batch runs across many agencies.

How do I handle a service that is both added and removed on the same date?

Apply removals last, so the date ends up removed. The feed is contradictory and should be reported, but declining to promise service is a smaller failure than advertising a trip that will not run.

Can I cache the expanded map between runs?

Yes, and you should. The map is a pure function of the feed archive, so key the cache on the feed’s content checksum — the same digest used for feed version control — and the cache invalidates itself whenever the agency republishes.