Cut pandas memory by 60–80% on GTFS feeds with three changes to pd.read_csv: pass usecols to load only the columns you need, supply an explicit dtype dict that downcasts numeric fields (Int8 for enumerated flags, float32 for coordinates), and set dtype_backend="pyarrow" so string identifiers become Arrow buffers instead of Python object arrays. No loss of data fidelity, no change to downstream API surface.

Why GTFS Feeds Trigger Memory Bloat

GTFS datasets are structurally repetitive but numerically dense. A single metropolitan export from stop_times.txt can contain tens of millions of rows, each storing time strings, sequence integers, and foreign-key identifiers that recur thousands of times. When pd.read_csv() runs without explicit instructions it defaults to three expensive choices:

• Promoting every column that contains at least one NaN to float64, even when the underlying values are small integers like pickup_type (range 0–3)
• Allocating a separate Python str object for every cell in string columns, producing one Python object per stop_id repetition across millions of rows
• Materialising every column in the CSV regardless of whether your pipeline needs it

These defaults misalign with transit data characteristics. Transit identifiers (route_id, trip_id, stop_id) are categorical: a finite vocabulary repeated millions of times. Coordinates are bounded floats that never require 64-bit precision. Time fields follow the strict HH:MM:SS format defined by the GTFS specification. Forcing pandas to recognise these patterns at read time prevents the exponential memory scaling that causes pipeline failures as feeds grow.

The diagram below shows where each optimisation intercepts the default allocation path.

Root Cause Analysis

Three independent mechanisms interact to inflate memory:

Silent type promotion. Pandas infers column types from a sample of rows. If any sampled row in stop_sequence is missing, the inferred type becomes float64 even when 99.9% of values are small integers. The GTFS spec marks several fields as conditionally required, so real agency feeds regularly produce sparse columns that trigger this promotion on every load.

Python object overhead for string columns. A Python str object carries roughly 50 bytes of interpreter overhead regardless of its payload. An object-dtype column holding one million stop_id strings of average length 8 characters therefore consumes around 50 MB in CPython, where a compact Arrow string buffer would use under 10 MB.

Loading unused optional fields. GTFS defines dozens of optional columns (wheelchair_boarding, platform_code, stop_timezone, stop_url). When pd.read_csv loads a file without a usecols filter, every optional column allocates memory whether or not the pipeline references it.

For a complete treatment of how these issues compound at feed scale, see the memory-efficient processing for large feeds guide, which covers chunked ingestion and Parquet persistence on top of the per-file dtype techniques described here.

Production-Ready Python Implementation

The following script loads stops.txt and stop_times.txt with production-ready memory constraints, converts HH:MM:SS time strings to integer seconds for downstream routing calculations, and reports the actual memory footprint with and without optimisation. Every variable name matches realistic GTFS column names; no toy placeholders are used.

import logging
import os

import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger(__name__)

# ---------------------------------------------------------------------------
# Column selection: only fields required for routing analysis
# ---------------------------------------------------------------------------
STOPS_COLS = [
    "stop_id", "stop_name", "stop_lat", "stop_lon",
    "location_type", "parent_station",
]
STOP_TIMES_COLS = [
    "trip_id", "arrival_time", "departure_time",
    "stop_id", "stop_sequence", "pickup_type", "drop_off_type",
]

# ---------------------------------------------------------------------------
# Explicit dtype overrides applied on top of the pyarrow backend defaults.
# ID and string columns (stop_id, trip_id, stop_name, arrival_time, etc.)
# are left to pyarrow's large_utf8 — equivalent to string[pyarrow].
# Numeric columns are downcast to the smallest safe type.
# ---------------------------------------------------------------------------
STOPS_NUMERIC_DTYPE: dict[str, str] = {
    "stop_lat":      "float32",   # WGS84 lat; float32 gives ~1 cm precision
    "stop_lon":      "float32",   # WGS84 lon
    "location_type": "Int8",      # nullable int; 0–4 per GTFS spec
}
STOP_TIMES_NUMERIC_DTYPE: dict[str, str] = {
    "stop_sequence": "Int16",     # typically < 32 767; nullable for safety
    "pickup_type":   "Int8",      # 0–3 per spec; NaN when field is absent
    "drop_off_type": "Int8",
}


def load_gtfs_tables(feed_dir: str) -> tuple[pd.DataFrame, pd.DataFrame]:
    """
    Load stops.txt and stop_times.txt with minimised memory footprint.

    Returns
    -------
    stops_df      : DataFrame indexed on stop_id
    stop_times_df : DataFrame with arrival_secs and departure_secs columns added
    """
    stops_path = os.path.join(feed_dir, "stops.txt")
    stop_times_path = os.path.join(feed_dir, "stop_times.txt")

    # dtype_backend="pyarrow" maps all inferred columns to Arrow types,
    # eliminating Python object arrays for string columns.
    # Per-column dtype overrides further downcast numeric fields.
    stops_df = pd.read_csv(
        stops_path,
        usecols=STOPS_COLS,
        dtype=STOPS_NUMERIC_DTYPE,
        dtype_backend="pyarrow",
        encoding="utf-8-sig",   # strips BOM from agency exports
    )
    logger.info(
        "stops.txt loaded: %s rows, %.2f MB",
        len(stops_df),
        stops_df.memory_usage(deep=True).sum() / 1_048_576,
    )

    stop_times_df = pd.read_csv(
        stop_times_path,
        usecols=STOP_TIMES_COLS,
        dtype=STOP_TIMES_NUMERIC_DTYPE,
        dtype_backend="pyarrow",
        encoding="utf-8-sig",
    )
    logger.info(
        "stop_times.txt loaded: %s rows, %.2f MB",
        len(stop_times_df),
        stop_times_df.memory_usage(deep=True).sum() / 1_048_576,
    )

    return stops_df, stop_times_df


def convert_gtfs_time_to_seconds(time_series: pd.Series) -> pd.Series:
    """
    Convert a Series of HH:MM:SS strings to integer seconds past midnight.

    Handles values above 24:00:00 (e.g. '25:30:00') that represent valid
    GTFS overnight service departing after calendar midnight.
    """
    parts = time_series.str.split(":", expand=True)
    return (
        parts[0].astype("Int32") * 3600
        + parts[1].astype("Int32") * 60
        + parts[2].astype("Int32")
    )


if __name__ == "__main__":
    feed_directory = "./sample_gtfs"

    stops_df, stop_times_df = load_gtfs_tables(feed_directory)

    # Add integer-second columns for routing engines (Dijkstra, raptor, etc.)
    stop_times_df["arrival_secs"]   = convert_gtfs_time_to_seconds(stop_times_df["arrival_time"])
    stop_times_df["departure_secs"] = convert_gtfs_time_to_seconds(stop_times_df["departure_time"])

    # Verify memory footprint
    stops_mb      = stops_df.memory_usage(deep=True).sum() / 1_048_576
    stop_times_mb = stop_times_df.memory_usage(deep=True).sum() / 1_048_576

    print(f"stops.txt      : {stops_mb:.2f} MB  ({len(stops_df):,} rows)")
    print(f"stop_times.txt : {stop_times_mb:.2f} MB  ({len(stop_times_df):,} rows)")

Step-by-Step Walkthrough

usecols=STOPS_COLS and usecols=STOP_TIMES_COLS. Pandas passes the column filter to the CSV tokenizer before memory is allocated. Any column not listed is never materialised — not read into a Python object, not type-inferred, not stored. This is the cheapest possible reduction: unused columns cost exactly zero bytes. Always audit the GTFS spec to determine which optional fields your downstream consumers actually need.

dtype=STOPS_NUMERIC_DTYPE with nullable integer types. Specifying "Int8" (capital I) rather than "int8" tells pandas to use the pd.Int8Dtype extension, which stores a separate boolean mask for NaN positions. Without this, a single missing value in location_type forces the entire column to float64 — eight bytes per row instead of one. For a feed with 50,000 stops, that difference is negligible; for stop_times.txt with 20 million rows and a nullable pickup_type column, it saves 140 MB.

dtype_backend="pyarrow". This flag, introduced in pandas 2.0, routes type inference through Apache Arrow’s type system rather than NumPy. String columns become large_utf8 (Arrow’s native string type), which stores all strings in a contiguous byte buffer with an offset array — eliminating one Python str object per cell. Arrow also handles null values without type promotion, so the dtype overrides above interact cleanly with it: Arrow nullable integers need no separate mask allocation beyond what Arrow already provides.

encoding="utf-8-sig". Some transit agency export toolchains prepend a UTF-8 byte-order mark to CSV files. Without this encoding flag, the BOM character becomes part of the first column name, silently breaking every dtype lookup and usecols filter that references that column by name.

convert_gtfs_time_to_seconds. The function splits on : and casts each part to Int32 before arithmetic. Using Int32 rather than Python int keeps the computation inside pandas’ vectorised engine. The result column (arrival_secs, departure_secs) is an Int32 array of seconds past midnight, where values above 86,400 represent valid overnight departures. This integer representation is the expected input for most open-source routing engines and avoids the ValueError that pd.to_datetime raises when it encounters times like 25:30:00.

Verification and Output

Verify memory savings before integrating this loader into a production pipeline. The key discipline is always passing deep=True to memory_usage:

import pandas as pd

def compare_dtype_strategies(csv_path: str, usecols: list[str], dtype_override: dict) -> None:
    """
    Print a memory comparison between default and optimised pd.read_csv calls.
    deep=True is mandatory for string columns — without it, object columns
    report only pointer size (8 bytes/element), not actual string bytes.
    """
    default_df = pd.read_csv(csv_path, usecols=usecols)
    optimised_df = pd.read_csv(
        csv_path,
        usecols=usecols,
        dtype=dtype_override,
        dtype_backend="pyarrow",
        encoding="utf-8-sig",
    )

    default_mb   = default_df.memory_usage(deep=True).sum()   / 1_048_576
    optimised_mb = optimised_df.memory_usage(deep=True).sum() / 1_048_576
    saving_pct   = 100 * (1 - optimised_mb / default_mb)

    print(f"Default   : {default_mb:.2f} MB")
    print(f"Optimised : {optimised_mb:.2f} MB")
    print(f"Reduction : {saving_pct:.1f}%")

    # Spot-check dtypes to confirm overrides applied
    for col, expected_dtype in dtype_override.items():
        if col in optimised_df.columns:
            actual = str(optimised_df[col].dtype)
            status = "OK" if expected_dtype.lower() in actual.lower() else "MISMATCH"
            print(f"  {col:<30} expected={expected_dtype}  actual={actual}  [{status}]")

Expected output for a 5-million-row stop_times.txt with default settings versus the optimised load:

Default   : 312.40 MB
Optimised : 68.15 MB
Reduction : 78.2%

  stop_sequence                  expected=Int16   actual=int16[pyarrow]  [OK]
  pickup_type                    expected=Int8    actual=int8[pyarrow]   [OK]
  drop_off_type                  expected=Int8    actual=int8[pyarrow]   [OK]

Per-column savings typically follow this pattern, confirming the strategy works as intended:

Gotchas and Edge Cases

• GTFS times above 23:59:59 break pd.to_datetime. The spec allows values like 25:30:00 for service crossing calendar midnight. Never parse arrival_time or departure_time with pd.to_datetime — ingest them as "string" dtype and use convert_gtfs_time_to_seconds or pd.to_timedelta for arithmetic. Applying timezone normalization to these values must happen after seconds conversion, not before.

• category dtype and cross-DataFrame merges produce silent empty results. When trip_id is "category" in both stop_times_df and trips_df, a pd.merge only matches rows where both sides share identical category dictionaries. Cast both sides to str before merging: chunk["trip_id"] = chunk["trip_id"].astype(str). After merging, re-cast back to "category" if memory budget requires it.

• PyArrow backend changes column dtype string representation. optimised_df["stop_sequence"].dtype returns int16[pyarrow], not Int16. Any downstream assert df[col].dtype == "Int16" test will fail. Update dtype assertions to use str(df[col].dtype) substring checks or isinstance(df[col].dtype, pd.ArrowDtype).

• shape_dist_traveled is often an empty string, not NaN. Agencies that do not populate this optional column frequently emit it as an empty field rather than a missing value. Specifying dtype={"shape_dist_traveled": "float32"} will raise a ValueError on empty strings. Use "Float32" (capital F, pandas nullable float) or exclude the column from usecols and cast after a pd.to_numeric(..., errors="coerce") call.

Frequently asked questions

Related

• Memory-Efficient Processing for Large Feeds — chunked ingestion, incremental joins, and Parquet persistence for feeds that exceed single-node RAM
• Handling Frequency-Based vs Timetable Schedules — time normalization workflows including frequencies.txt expansion and overnight trip handling
• Mastering stops.txt and stop_times.txt Relationships — GTFS relational model, foreign-key constraints, and referential integrity checks

Up: Memory-Efficient Processing for Large Feeds | Python Parsing & Data Normalization | Home