Step-by-Step Guide to Parsing GTFS with Partridge

Q: Why does Partridge return an empty DataFrame instead of raising an error for missing GTFS files?

Partridge returns an empty DataFrame with the correct column schema when an optional GTFS file is absent. This prevents pipeline crashes on incomplete feeds but can mask data quality issues — always check shape and column presence after loading.

Q: How does Partridge handle GTFS times over 24:00:00?

Partridge preserves the raw HH:MM:SS string from stop_times.txt. When you convert with pd.to_timedelta(), values like '25:30:00' become '1 day 01:30:00', so adding to a base Timestamp automatically places overnight trips on the correct calendar day.

Q: What is the correct Partridge view filter to isolate a single route?

Filter trips.txt by both service_id and route_id: view = {'trips.txt': {'service_id': active_ids, 'route_id': {'target_route_id'}}}. Partridge propagates this filter to stop_times.txt automatically.

Call ptg.load_feed(feed_path, view={"trips.txt": {"service_id": active_ids}}) where active_ids comes from ptg.read_service_ids_by_date(feed_path)[target_date]. This applies a lazy row filter before any DataFrame materializes, so only trips running on your target date are loaded — eliminating the need to post-filter millions of stop_times.txt rows by hand.

Root cause: why this parsing problem is harder than it looks

A GTFS ZIP looks like a flat folder of CSV files, but three subtleties break naive pd.read_csv() pipelines before you reach the first join:

Calendar branching. Every feed bundles multiple service calendars — weekday, weekend, holiday, and exception entries from calendar_dates.txt. Iterating stop_times.txt without first isolating the correct service_id set means your DataFrame silently mixes trips from different operating days, producing departure times that never ran on the date you intended.

Extended midnight arithmetic. The GTFS specification encodes departure times as HH:MM:SS offsets from the start of the service day, not wall-clock times. Buses that continue past midnight carry values like 25:30:00 — a valid GTFS time that pd.read_datetime() and pd.to_datetime() reject outright, causing silent NaT propagation or ValueErrors mid-pipeline.

Referential integrity across files. trips.txt, stop_times.txt, and calendar.txt form a three-way join. Real agency feeds routinely ship trip_id values in stop_times.txt that have no matching row in trips.txt, and service_id values in trips.txt that neither calendar.txt nor calendar_dates.txt defines. Partridge enforces these foreign-key relationships during load and prunes orphaned rows automatically.

Production-ready implementation

The block below is a single copy-pasteable module. It resolves active service IDs, loads a filtered feed, applies timezone normalization using the agency’s own agency_timezone field, asserts referential integrity, and writes Parquet output.

"""gtfs_partridge_parse.py — production GTFS loader using partridge + pandas."""

from __future__ import annotations

import logging
from datetime import date, timedelta
from pathlib import Path

import pandas as pd
import partridge as ptg

log = logging.getLogger(__name__)


def load_active_feed(
    feed_path: str | Path,
    target_date: date | None = None,
) -> dict[str, pd.DataFrame]:
    """Return a dict of filtered DataFrames for *target_date* (default: tomorrow).

    Keys: 'stops', 'routes', 'trips', 'stop_times', 'agency'
    All departure/arrival columns are timezone-aware pd.Timestamp values.
    """
    feed_path = Path(feed_path)
    if not feed_path.exists():
        raise FileNotFoundError(f"GTFS archive not found: {feed_path}")

    target_date = target_date or date.today() + timedelta(days=1)
    log.info("Resolving service IDs for %s from %s", target_date, feed_path.name)

    # --- Step 1: resolve active service IDs -----------------------------------
    service_ids_by_date: dict[date, frozenset[str]] = ptg.read_service_ids_by_date(
        str(feed_path)
    )
    active_ids: frozenset[str] = service_ids_by_date.get(target_date, frozenset())

    if not active_ids:
        raise ValueError(
            f"No active service found for {target_date}. "
            f"Feed covers {min(service_ids_by_date)} – {max(service_ids_by_date)}."
        )
    log.info("Active service IDs: %s", active_ids)

    # --- Step 2: load a filtered Feed object ----------------------------------
    # The view dict prunes trips.txt before any DataFrame is materialized.
    # Partridge propagates the filter to stop_times.txt, calendar.txt, etc.
    view = {"trips.txt": {"service_id": active_ids}}
    feed: ptg.Feed = ptg.load_feed(str(feed_path), view=view)

    # --- Step 3: extract DataFrames with explicit dtypes ----------------------
    agency: pd.DataFrame = feed.agency.astype(
        {"agency_id": "string", "agency_timezone": "string"}
    )
    stops: pd.DataFrame = feed.stops.astype(
        {
            "stop_id": "string",
            "stop_name": "string",
            "stop_lat": "float64",
            "stop_lon": "float64",
        }
    )
    routes: pd.DataFrame = feed.routes.astype(
        {"route_id": "string", "route_short_name": "string", "route_type": "int8"}
    )
    trips: pd.DataFrame = feed.trips.astype(
        {"trip_id": "string", "route_id": "string", "service_id": "string"}
    )
    stop_times: pd.DataFrame = feed.stop_times.astype(
        {
            "trip_id": "string",
            "stop_id": "string",
            "stop_sequence": "int32",
            "departure_time": "string",
            "arrival_time": "string",
        }
    )

    log.info(
        "Loaded: %d trips, %d stop_time rows, %d stops",
        len(trips),
        len(stop_times),
        len(stops),
    )

    # --- Step 4: normalize GTFS time strings to timezone-aware Timestamps ----
    # pd.to_timedelta handles values >= 24:00:00 (overnight trips) correctly:
    # "25:30:00" becomes Timedelta("1 day 01:30:00"), placing the event on
    # the next calendar day when added to base_ts.
    tz: str = agency["agency_timezone"].iloc[0]
    base_ts = pd.Timestamp(target_date, tz=tz)

    for col in ("departure_time", "arrival_time"):
        td_col = col.replace("_time", "_td")
        stop_times[td_col] = pd.to_timedelta(stop_times[col], errors="coerce")
        stop_times[col.replace("_time", "_dt")] = base_ts + stop_times[td_col]

    # Drop intermediate timedelta columns
    stop_times.drop(columns=["departure_td", "arrival_td"], inplace=True)

    return {
        "agency": agency,
        "stops": stops,
        "routes": routes,
        "trips": trips,
        "stop_times": stop_times,
    }


def validate_and_export(tables: dict[str, pd.DataFrame], output_dir: str | Path) -> None:
    """Assert referential integrity and write Parquet files."""
    output_dir = Path(output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)

    trips = tables["trips"]
    stop_times = tables["stop_times"]

    # Referential integrity: every trip_id in stop_times must exist in trips
    orphan_trips = set(stop_times["trip_id"].unique()) - set(trips["trip_id"].unique())
    if orphan_trips:
        log.warning("Orphaned trip_ids in stop_times (pruned): %d", len(orphan_trips))
        stop_times = stop_times[~stop_times["trip_id"].isin(orphan_trips)]

    assert (
        stop_times["departure_dt"].notna().all()
    ), "NaT values in departure_dt — check for malformed HH:MM:SS strings"

    for name, df in tables.items():
        out_path = output_dir / f"{name}.parquet"
        df.to_parquet(out_path, index=False)
        log.info("Wrote %s (%d rows) → %s", name, len(df), out_path)

Step-by-step walkthrough

The implementation follows six discrete stages, each isolating a distinct failure mode.

Stage 1 — Resolve service IDs. ptg.read_service_ids_by_date() reads only calendar.txt and calendar_dates.txt from the ZIP, merging added/removed exception entries from calendar_dates.txt on top of the base calendar. The result is a dict[date, frozenset[str]]. Checking the date range of this dict surfaces feeds that expired before your target date — a common issue with cached or stale downloads.

Stage 2 — Build the view filter. The view dictionary maps each GTFS filename to a column-value filter. Passing {"trips.txt": {"service_id": active_ids}} tells Partridge to discard any row in trips.txt whose service_id is not in active_ids before allocating a DataFrame. Because stop_times.txt joins through trip_id, Partridge propagates this constraint automatically.

Stage 3 — Load the Feed (lazy). ptg.load_feed() extracts the ZIP and builds an internal map of file handles, but does not read CSV data into memory yet. DataFrames are only materialized when you access a property (feed.stop_times, etc.). This means the filter has already been applied at the file level before any pandas allocation occurs, keeping peak RSS well below the uncompressed feed size.

Stage 4 — Extract with explicit dtypes. Partridge returns DataFrames with mixed-inferred dtypes. Casting immediately after extraction — "stop_id": "string", "stop_lat": "float64", "route_type": "int8" — prevents downstream type coercion surprises and reduces memory footprint. The pandas "string" dtype backed by StringArray avoids the object dtype’s per-element Python overhead.

Stage 5 — Normalize GTFS time strings. pd.to_timedelta() is the correct tool for GTFS HH:MM:SS strings because it handles values above "23:59:59" without error. Adding the resulting Timedelta to a timezone-aware pd.Timestamp anchored to the service date produces fully localized Timestamp values. Overnight trips (values like "25:30:00") automatically land on the next calendar day. See converting local transit times to UTC for the full DST-safe variant.

Stage 6 — Validate and export. The assert on departure_dt.notna() catches any unparseable time strings (malformed HH:MM:SS, empty cells) before they silently corrupt a routing engine. Parquet output via pyarrow preserves pd.StringDtype and timezone-aware DatetimeTZDtype without lossy conversion.

Verification and output

Run these checks immediately after load_active_feed() returns to confirm filter integrity:

tables = load_active_feed("agency-gtfs.zip", target_date=date(2026, 6, 25))

trips      = tables["trips"]
stop_times = tables["stop_times"]
stops      = tables["stops"]

# 1. Every trip in stop_times resolves to a known trip
assert set(stop_times["trip_id"].unique()).issubset(set(trips["trip_id"].unique())), \
    "Orphaned trip_ids in stop_times"

# 2. No NaT departure timestamps
assert stop_times["departure_dt"].notna().all(), \
    "NaT in departure_dt — malformed time strings present"

# 3. Stop references are valid
assert set(stop_times["stop_id"].unique()).issubset(set(stops["stop_id"].unique())), \
    "stop_times references stop_ids absent from stops.txt"

# 4. Inspect shape
print(f"Trips: {len(trips)}")
print(f"Stop-time rows: {len(stop_times)}")
print(f"Departure range: {stop_times['departure_dt'].min()} → {stop_times['departure_dt'].max()}")
print(stop_times[["trip_id", "stop_id", "stop_sequence", "departure_dt"]].head())

Expected output for a mid-sized urban feed on a weekday:

Trips: 2814
Stop-time rows: 87 342
Departure range: 2026-06-25 04:30:00-05:00 → 2026-06-26 02:15:00-05:00
   trip_id stop_id  stop_sequence              departure_dt
0  T100001   S4821              1 2026-06-25 05:10:00-05:00
1  T100001   S4930              2 2026-06-25 05:14:00-05:00

The departure range spanning into the next calendar day confirms overnight trips are handled correctly — no extra timedelta(days=1) offset is needed.

Gotchas and edge cases

calendar_dates.txt-only feeds. Some agencies (especially US commuter rails) omit calendar.txt entirely and define every service day as an explicit ADDED exception in calendar_dates.txt. ptg.read_service_ids_by_date() handles this correctly, but if you call Partridge’s lower-level CSV readers directly you must check both files.
Reused stop_id values across agencies. When you merge feeds from multiple agencies into a single DataFrame, stop_id collisions are common — agencies independently use short numeric IDs like "1" or "100". Prefix each stop_id with an agency_id column before any cross-agency join. The batch processing patterns for multi-agency feeds page covers a namespace-safe merge strategy.
Feeds with no active service on the target date. ptg.read_service_ids_by_date() returns an empty frozenset — not a KeyError — when the target date is outside the feed’s validity window. Always check if not active_ids and surface a descriptive error that includes the feed’s actual date range, otherwise ptg.load_feed() will silently return empty DataFrames for all tables.
pandas 2.2+ FutureWarning on dtype inference. Partridge 1.1.0 passes no explicit dtype argument to pd.read_csv(), which triggers a FutureWarning in pandas 2.2 about future dtype inference changes. Suppress with warnings.filterwarnings("ignore", category=FutureWarning, module="partridge") or pin pandas<2.2 until upstream resolves it. Track this in requirements.txt rather than silencing globally.

Frequently asked questions

Why does Partridge return an empty DataFrame instead of raising an error for missing GTFS files?

Partridge returns an empty DataFrame with the correct column schema when an optional GTFS file is absent from the ZIP. This prevents pipeline crashes on feeds that legitimately omit optional files (such as shapes.txt or frequencies.txt) but can mask genuine data quality issues — always check .shape and column presence immediately after loading, and treat a zero-row result as a signal to inspect the source archive.

How does Partridge handle GTFS times over 24:00:00?

Partridge preserves the raw HH:MM:SS string from stop_times.txt without modification. When you convert with pd.to_timedelta(), a value like "25:30:00" becomes Timedelta("1 day 01:30:00"). Adding that to a pd.Timestamp anchored to the service date places the event at 01:30 on the following calendar day — exactly what the GTFS specification intends for overnight trips that depart after the service-day boundary.

What is the correct Partridge view filter to isolate a single route?

Filter trips.txt by both service_id and route_id in the same view dict:

view = {
    "trips.txt": {
        "service_id": active_ids,
        "route_id": {"target_route_id"},
    }
}

Partridge propagates the resulting trip_id set down to stop_times.txt automatically, so you do not need a second filter on the stop-times file.

Parsing GTFS with Pandas and Partridge — parent page covering the full pipeline, batch scheduling, and incremental updates
Timezone Handling and Schedule Normalization — DST-safe UTC conversion for stop_times.txt departure values
Converting Local Transit Times to UTC in Python — worked example anchoring GTFS offsets to zoneinfo timezones
Memory-Efficient Processing for Large Feeds — chunked reading and Parquet persistence when feeds exceed 500 MB
Fixing Missing stop_times.txt Records in Python — diagnosing and repairing orphaned trip_id references
Python Parsing & Data Normalization — section overview
Home — transit-data.com