pandas vs polars vs partridge for GTFS

Reach for pandas when convenience and ecosystem breadth matter more than raw throughput, switch to polars when a multi-million-row stop_times.txt is straining memory or parse time, and pick partridge when you need the loader itself to enforce foreign-key integrity and slice the feed to a single service date. None of the three is universally best; the right answer depends on whether your bottleneck is developer time, machine resources, or referential correctness.

Where the Three Libraries Diverge

A GTFS feed is not one uniform dataset — it is a dozen CSV files whose sizes span four orders of magnitude. agency.txt holds a handful of rows; stop_times.txt for a national operator holds tens of millions. Any honest comparison has to acknowledge that the library that wins on stop_times.txt may be irrelevant for calendar.txt. The differences that matter concentrate almost entirely in how each tool ingests that one dominant file.

pandas reads CSV single-threaded into a NumPy-backed frame. String columns such as trip_id, stop_id, and arrival_time become Python object arrays, which are memory-hungry: every value is a boxed Python string with pointer overhead. This is why a 900 MB stop_times.txt can expand to several gigabytes of resident memory. The upside is that pandas is the lingua franca of the Python data stack — every downstream library, from geopandas to scikit-learn, speaks it natively.

polars reads CSV in parallel across cores into an Apache Arrow columnar buffer. Strings are stored contiguously rather than as boxed objects, so the same stop_times.txt occupies a fraction of the memory and parses in a fraction of the wall-clock time. Its lazy API (scan_csv) additionally lets you push filters and column projections down into the reader, so you never fully materialise columns you will drop. The cost is a smaller ecosystem and an API that pandas users must relearn.

partridge is not a general dataframe engine — it is a GTFS-aware loader built on top of pandas. It models the feed as a graph of foreign-key relationships (trips.trip_id → stop_times.trip_id, routes.route_id → trips.route_id, and so on) and, given a filter such as a service_id set for one calendar date, it loads only the rows reachable from that filter. The practical effect is that you read a slice of stop_times.txt instead of the whole thing, and every table you get back is guaranteed to be referentially consistent.

The table below summarises how the three tools behave on the file that decides most GTFS pipelines.

Production-Ready Python Implementation

The script below is a self-contained benchmark harness. It loads the same stop_times.txt three ways, records peak resident memory and wall-clock time for each, and prints a comparison table. It uses tracemalloc for allocation tracking and time.perf_counter for timing, with explicit dtype maps so that string identifier columns are never silently cast to integers.

import time
import tracemalloc
import logging
from pathlib import Path

import pandas as pd
import polars as pl
import partridge as ptg

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

# Explicit dtypes keep zero-padded ids like "0012" as strings, not integers.
PANDAS_DTYPES = {
    "trip_id": str,
    "stop_id": str,
    "arrival_time": str,     # GTFS times can exceed 24:00:00
    "departure_time": str,
    "stop_sequence": "Int64",
}

POLARS_SCHEMA = {
    "trip_id": pl.Utf8,
    "stop_id": pl.Utf8,
    "arrival_time": pl.Utf8,
    "departure_time": pl.Utf8,
    "stop_sequence": pl.Int64,
}


def _measure(label: str, fn) -> dict:
    """Run fn() once, returning wall-clock seconds, peak MiB, and row count."""
    tracemalloc.start()
    start = time.perf_counter()
    row_count = fn()
    elapsed = time.perf_counter() - start
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    result = {"library": label, "seconds": elapsed, "peak_mib": peak / 2**20, "rows": row_count}
    logging.info("%-9s | %7.2f s | %8.1f MiB | %d rows",
                 label, result["seconds"], result["peak_mib"], row_count)
    return result


def bench_pandas(feed_dir: Path) -> int:
    stop_times = pd.read_csv(feed_dir / "stop_times.txt", dtype=PANDAS_DTYPES)
    # Representative workload: trips serving each stop.
    _ = stop_times.groupby("stop_id", sort=False)["trip_id"].nunique()
    return len(stop_times)


def bench_polars(feed_dir: Path) -> int:
    # Lazy scan pushes the projection into the reader.
    lazy = pl.scan_csv(feed_dir / "stop_times.txt", schema_overrides=POLARS_SCHEMA)
    stop_times = lazy.select(["trip_id", "stop_id", "stop_sequence"]).collect()
    _ = stop_times.group_by("stop_id").agg(pl.col("trip_id").n_unique())
    return stop_times.height


def bench_partridge(feed_dir: Path) -> int:
    # partridge slices the feed to the busiest service date and prunes
    # every table to rows reachable from that service_id set.
    feed_path = str(feed_dir)
    busiest_date, service_ids = ptg.read_busiest_date(feed_path)
    view = {"trips.txt": {"service_id": service_ids}}
    feed = ptg.load_feed(feed_path, view=view)
    stop_times = feed.stop_times          # already FK-filtered to that date
    _ = stop_times.groupby("stop_id", sort=False)["trip_id"].nunique()
    logging.info("partridge sliced to %s (%d service_ids)", busiest_date, len(service_ids))
    return len(stop_times)


def run_benchmark(feed_dir: str) -> pd.DataFrame:
    feed_path = Path(feed_dir)
    if not (feed_path / "stop_times.txt").exists():
        raise FileNotFoundError(f"No stop_times.txt under {feed_path}")

    results = [
        _measure("pandas", lambda: bench_pandas(feed_path)),
        _measure("polars", lambda: bench_polars(feed_path)),
        _measure("partridge", lambda: bench_partridge(feed_path)),
    ]
    summary = pd.DataFrame(results).sort_values("seconds").reset_index(drop=True)
    return summary


if __name__ == "__main__":
    report = run_benchmark("/data/gtfs/national")
    print(report.to_string(index=False))

Step-by-Step Walkthrough

Isolated measurement per library. Each parse runs inside _measure, which brackets the call with tracemalloc and perf_counter. Because tracemalloc tracks Python-level allocations, it captures the object-array overhead that makes pandas string columns expensive and shows how much smaller the polars Arrow buffer is for the identical file.

Honest workloads, not just reads. Every function performs the same representative aggregation — counting distinct trip_id values per stop_id — after loading. A parse benchmark that never touches the data understates pandas’ cost, because the boxed-string penalty is paid during the groupby, not only at read time.

polars pushes work into the reader. pl.scan_csv(...).select([...]).collect() is lazy: the projection to three columns is pushed down so the CSV reader never fully decodes arrival_time or departure_time. This is why polars wins hardest on wide files where you only need a few columns.

partridge changes the row count, not just the speed. read_busiest_date finds the service date with the most trips and returns its service_id set; the view then prunes stop_times.txt to only those trips during load. The rows figure it reports is therefore smaller than the raw file — that is the whole point. If your task genuinely needs one service day, partridge does less work because it reads less data, and every returned table is referentially clean.

Explicit dtypes everywhere. Both the pandas dtype map and the polars schema_overrides pin trip_id and stop_id to strings. Without this, a feed whose identifiers happen to be all-numeric will silently load them as integers, dropping leading zeros and breaking later joins.

Verification and Output

Run the harness against a real feed and assert that the three loaders agree on the shape of the data they claim to load, then confirm the performance ordering you expect.

report = run_benchmark("/data/gtfs/national")

# polars should never use more peak memory than pandas on stop_times.txt
pandas_peak = report.loc[report["library"] == "pandas", "peak_mib"].iloc[0]
polars_peak = report.loc[report["library"] == "polars", "peak_mib"].iloc[0]
assert polars_peak <= pandas_peak, "Unexpected: polars used more memory than pandas"

# partridge reads a filtered slice, so its row count must not exceed pandas' full read
pandas_rows = report.loc[report["library"] == "pandas", "rows"].iloc[0]
partridge_rows = report.loc[report["library"] == "partridge", "rows"].iloc[0]
assert partridge_rows <= pandas_rows, "partridge slice larger than full feed — check the view filter"

print("Benchmark assertions passed")

On a national feed with roughly twelve million stop_times.txt rows, a representative run prints wall-clock times in the order polars < partridge < pandas, and peak memory in the order partridge < polars < pandas. The exact figures depend on core count and disk speed, but the ordering is stable: polars wins on full-feed throughput, partridge wins on memory whenever the task is date-scoped, and pandas trails on both while remaining the easiest to integrate.

Gotchas and Edge Cases

• polars parses GTFS times as strings, not durations. arrival_time values such as 25:30:00 are valid GTFS but not valid HH:MM:SS clock times. Keep them as Utf8 and convert to seconds-since-midnight arithmetically; letting polars infer a time dtype will drop every overnight departure. The same caution applies when you later convert to UTC for schedule normalization.

• partridge’s filter is only as good as your calendar logic. read_busiest_date picks one date; if your analysis needs a full week or a holiday exception in calendar_dates.txt, build the service_id set yourself and pass it in the view. A too-narrow filter silently omits valid service.

• Memory numbers depend on the string cache. polars deduplicates repeated strings, so a feed with a few thousand distinct stop_id values repeated across millions of rows compresses far better than one with high-cardinality identifiers. Benchmark on your own feed rather than trusting a headline ratio; see the memory-efficient processing guide for the underlying mechanics.

• Converting polars back to pandas can erase the savings. If a downstream step calls .to_pandas() on a large frame, you re-materialise the boxed-string arrays and pay the pandas memory cost anyway. Keep the heavy aggregation in polars and only convert the small result.

Frequently Asked Questions

Related

• Step-by-Step Guide to Parsing GTFS with Partridge — how the foreign-key-aware loader prunes a feed to one service date
• Optimizing Pandas Memory Usage for Transit Feeds — categoricals and downcasting when you must stay in pandas
• Memory-Efficient Processing for Large Feeds — chunking and columnar formats for feeds that exceed RAM
• Up: Parsing GTFS with pandas and partridge — the loader patterns these benchmarks build on
• Section: Python Parsing & Data Normalization — the full GTFS parsing and normalization pipeline · Home