1. Home
  2. Python Parsing Data Normalization
  3. Memory Efficient Processing For Large Feeds
  4. Chunked Reading Of Stop Times In Pandas

Chunked Reading of stop_times in pandas

Chunking only helps if each chunk is reduced before it is released. Reading stop_times.txt in pieces and concatenating them at the end uses more memory than reading it whole, because the full table is rebuilt and the per-chunk copies sit alongside it. Read with a fixed chunksize and an explicit dtype schema, aggregate each chunk down to what you actually need, carry any trailing trip fragment across the boundary so no trip is processed in halves, and unify categorical dtypes before combining anything. The strategy ladder this sits on is in memory-efficient processing for large feeds.

Chunking that helps and chunking that does not A grid over three chunked implementations and what each does to peak memory. Peak memory Verdict Read whole, five columns 1.4 GB the baseline Chunk, then concatenate 1.6 GB worse than not chunking Chunk, reduce per chunk 210 MB what chunking is for

Root Cause Analysis

stop_times.txt is where GTFS memory problems live. It carries one row per call at a stop, so a metropolitan feed holds around 1.8 million rows against 31,000 trips and 8,900 stops — roughly 87% of the feed’s rows in one file. Read naively it can occupy well over a gigabyte, and that is before anything is computed.

Chunking is the standard response and it is easy to get wrong in three specific ways.

Concatenating at the end. pd.concat(list(reader)) reads in chunks and then reassembles the whole table, so peak memory is the full table plus whatever the chunks cost to build. This is strictly worse than a single read and it is the most common implementation.

Splitting a trip. A chunk boundary lands wherever the row count says, which is almost always in the middle of a trip’s ordered rows. Any per-trip computation — first and last stop, whether times increase, interpolating an untimed call between two timepoints — sees an incomplete trip in both chunks. The results are wrong for one trip per boundary, which on a hundred chunks is a hundred quietly broken trips.

Inconsistent categoricals. Casting trip_id to category per chunk builds a category list from that chunk’s values. Two chunks then have different category sets, and combining them either falls back to object — losing the entire memory saving — or, in older pandas, maps integer codes against the wrong list. The corruption is silent and total.

There is also a boundary worth naming: chunking is for computations that reduce. A global sort, a join against another large table, or repeated queries over the same data are not reductions, and hand-rolling them over chunks reimplements a query engine badly.

Production-Ready Python Implementation

python 
"""Process a large GTFS stop_times.txt in chunks without ever holding it whole."""
from __future__ import annotations

import logging
from dataclasses import dataclass, field

import pandas as pd

log = logging.getLogger("gtfs.chunked")

CHUNK_ROWS = 500_000
USE_COLUMNS = ["trip_id", "stop_id", "stop_sequence", "arrival_time", "departure_time"]
READ_DTYPES = {c: "string" for c in USE_COLUMNS}


@dataclass
class TripSummary:
    """The per-trip reduction — a few hundred bytes where the rows were megabytes."""
    first_stop: str
    last_stop: str
    first_departure_s: int
    last_arrival_s: int
    calls: int
    monotonic: bool


@dataclass
class ChunkedResult:
    trips: dict[str, TripSummary] = field(default_factory=dict)
    calls_per_stop: pd.Series = field(default_factory=lambda: pd.Series(dtype="int64"))
    chunks: int = 0
    rows: int = 0
    carried: int = 0          # rows carried across a boundary


def _seconds(series: pd.Series) -> pd.Series:
    parts = series.str.strip().str.split(":", expand=True)
    if parts.shape[1] < 3:
        return pd.Series([pd.NA] * len(series), dtype="Int64")
    numeric = parts.astype("Float64")
    return (numeric[0] * 3600 + numeric[1] * 60 + numeric[2]).astype("Int64")


def _summarise(group: pd.DataFrame) -> TripSummary:
    ordered = group.sort_values("stop_sequence")
    departures = ordered["departure_s"].dropna()
    return TripSummary(
        first_stop=str(ordered["stop_id"].iloc[0]),
        last_stop=str(ordered["stop_id"].iloc[-1]),
        first_departure_s=int(departures.iloc[0]) if len(departures) else -1,
        last_arrival_s=int(ordered["arrival_s"].dropna().iloc[-1])
        if ordered["arrival_s"].notna().any() else -1,
        calls=len(ordered),
        monotonic=bool(departures.is_monotonic_increasing),
    )


def process_stop_times(handle, chunk_rows: int = CHUNK_ROWS) -> ChunkedResult:
    result = ChunkedResult()
    stop_counts: list[pd.Series] = []
    carry = pd.DataFrame()        # the trailing, possibly incomplete, trip

    reader = pd.read_csv(handle, usecols=USE_COLUMNS, dtype=READ_DTYPES,
                         encoding="utf-8-sig", keep_default_na=False,
                         na_values=[""], chunksize=chunk_rows)

    for raw in reader:
        result.chunks += 1
        result.rows += len(raw)

        chunk = pd.concat([carry, raw], ignore_index=True) if len(carry) else raw
        chunk["stop_sequence"] = pd.to_numeric(chunk["stop_sequence"],
                                               errors="coerce").astype("Int32")
        chunk["arrival_s"] = _seconds(chunk["arrival_time"])
        chunk["departure_s"] = _seconds(chunk["departure_time"])

        # The last trip in the chunk is probably cut in half — hold it back.
        last_trip = chunk["trip_id"].iloc[-1]
        incomplete = chunk["trip_id"] == last_trip
        carry = chunk[incomplete].copy()
        result.carried = len(carry)
        complete = chunk[~incomplete]

        if len(complete):
            for trip_id, group in complete.groupby("trip_id", observed=True):
                result.trips[str(trip_id)] = _summarise(group)
            stop_counts.append(complete.groupby("stop_id", observed=True).size())

        # `chunk` and `complete` go out of scope here; only the reductions survive.
        log.debug("chunk %d: %d row(s), %d trip(s) closed, %d carried",
                  result.chunks, len(raw), complete["trip_id"].nunique() if len(complete) else 0,
                  len(carry))

    if len(carry):
        for trip_id, group in carry.groupby("trip_id", observed=True):
            result.trips[str(trip_id)] = _summarise(group)
        stop_counts.append(carry.groupby("stop_id", observed=True).size())

    if stop_counts:
        # Sum the per-chunk counts; the index is plain text, so no category
        # mismatch is possible when they are combined.
        result.calls_per_stop = (pd.concat(stop_counts).groupby(level=0).sum()
                                 .sort_values(ascending=False))

    log.info("processed %d row(s) in %d chunk(s): %d trip(s), %d stop(s) called at",
             result.rows, result.chunks, len(result.trips), len(result.calls_per_stop))
    return result
Why a trip must not be split at a boundary A chunk boundary falls in the middle of a trip's ordered calls, so the trailing fragment is carried forward and summarised with the next chunk. chunk 1 starts row 1 trip complete closed boundary trip cut in half carried forward into chunk 2 summarising both halves separately gives two wrong the carry is bounded by the longest trip in the feed, not by the chunk size

Step-by-Step Walkthrough

usecols runs before anything else. Five columns out of nine or more, chosen at read time, so the discarded columns are never parsed or allocated. This is the cheapest optimisation available and it belongs first — everything after it operates on less data.

The carry buffer is keyed on the last trip_id in the chunk. Because stop_times.txt is grouped by trip in every real feed, holding back every row belonging to the final trip guarantees that trip is complete when it is finally summarised. It also means the carry is bounded by the longest trip in the feed — a few hundred rows — rather than growing.

Only reductions escape the loop. result.trips holds a small dataclass per trip and stop_counts holds one Series per chunk. The chunk frames themselves are rebound on each iteration and collected. Peak memory is one chunk plus the accumulated reductions, which for 1.8 million rows is roughly 200 MB rather than 1.4 GB.

observed=True on every groupby. With categorical keys, pandas otherwise produces a row for every unseen category, which on a chunk containing 3% of the feed’s stops means 97% empty groups. It costs nothing here and prevents a real surprise if the identifiers are later cast to category.

The final combine groups on the index and sums. pd.concat(stop_counts) produces a long Series with repeated stop identifiers, and grouping on the index level adds them. Because the index is plain text rather than categorical, there is no category set to unify — which is the deliberate answer to the third failure mode above.

The trailing carry is processed after the loop. The last chunk’s final trip is never followed by another chunk, so it must be summarised explicitly. Forgetting this drops exactly one trip, which is small enough to go unnoticed and wrong enough to matter.

Verification and Output

python 
def verify(result: ChunkedResult, expected_rows: int | None = None) -> None:
    assert result.chunks > 0, "nothing was read"
    if expected_rows is not None:
        assert result.rows == expected_rows, (
            f"read {result.rows} rows, expected {expected_rows}")

    total_calls = sum(s.calls for s in result.trips.values())
    assert total_calls == result.rows, (
        f"{result.rows} rows read but {total_calls} accounted for in trip summaries — "
        "a chunk boundary dropped or duplicated a trip")

    assert int(result.calls_per_stop.sum()) == result.rows, (
        "per-stop call counts do not sum to the row count")

    for trip_id, summary in result.trips.items():
        assert summary.calls >= 1, f"trip {trip_id} summarised with no calls"
        if summary.first_departure_s >= 0 and summary.last_arrival_s >= 0:
            assert summary.last_arrival_s >= summary.first_departure_s, (
                f"trip {trip_id} arrives before it departs — likely a trip split "
                "across a chunk boundary and summarised twice")

The first two assertions are the ones that catch boundary bugs, and they catch them completely: every row must appear in exactly one trip summary and in exactly one stop count. A trip processed in halves shows up as a count mismatch immediately, rather than as a subtly wrong first stop nobody checks.

Output on a large feed:

text 
INFO gtfs.chunked: processed 1840112 row(s) in 4 chunk(s): 31042 trip(s), 8871 stop(s) called at

Peak resident memory for that run is about 210 MB. Reading the same file whole with the same five columns and text dtypes peaks at roughly 1.4 GB, and reading it whole with inferred dtypes peaks higher still.

What is allowed to leave the loop Only reductions escape each iteration; the chunk frames themselves are rebound and collected, which is what keeps peak memory at one chunk. Per-trip summary a few hundred bytes per trip Per-stop counts one small Series per chunk The carry buffer one trip's rows, at most The chunk itself never — it is released each iteration

Gotchas and Edge Cases

  • Feeds not grouped by trip. The carry logic assumes a trip’s rows are contiguous, which every real feed satisfies but the specification does not require. If a feed interleaves trips, the carry holds only part of the last trip and the assertions above fire — which is the correct outcome, and the fix is to sort the file first or switch tools.
  • A chunk containing exactly one trip. complete is then empty and everything carries forward, so the carry grows. Bounded by the largest trip in the feed, so it is safe, but it means chunk_rows should comfortably exceed the longest trip’s call count.
  • Categorical casts inside the loop. Do not. Cast after aggregation, or use plain text as above. This is the single most common way chunked GTFS processing corrupts identifiers.
  • low_memory warnings. Passing an explicit dtype for every column removes them entirely. A DtypeWarning in a chunked read means some column is being inferred differently per chunk, which is the mixed-type failure in dtype enforcement.
  • When to stop. If the reduction needs the whole table at once — a global sort, a join against another million-row table, or the same data queried repeatedly — chunking is the wrong tool. Convert to partitioned Parquet and query it, or read with polars, rather than reimplementing a query engine over chunks.

Frequently Asked Questions

Does chunking help if I concatenate the chunks at the end?

No — that is the most common mistake. Concatenating every chunk rebuilds the whole table in memory and adds the per-chunk copies on top, so peak memory is higher than reading it in one go. Chunking only helps if each chunk is reduced before it is released.

What breaks when a trip spans two chunks?

Any per-trip computation: first and last stop, monotonic time checks, interpolation between timepoints. The chunk boundary falls in the middle of a trip’s rows, so both halves are incomplete. Carry the trailing fragment forward and process it with the next chunk.

Why do categorical dtypes corrupt data across chunks?

Because each chunk builds its own category list from the values it happens to contain. Concatenating two frames with different category sets falls back to object, or worse, maps codes against the wrong categories. Unify the categories explicitly before combining.

When should I stop chunking and use something else?

When the aggregation itself needs the whole table — a global sort, a join against another large table, or repeated queries. Parquet plus DuckDB or polars handles those directly and is simpler than a hand-rolled chunk pipeline.