1. Home
  2. GTFS Feed Architecture Fundamentals
  3. Understanding GTFS Static Feed Structure
  4. Reading GTFS Zip Archives In Python

Reading GTFS Zip Archives in Python

Open the archive once, build a map from lowercased basename to member so directory prefixes and casing differences stop mattering, check the declared uncompressed sizes before reading anything, and then stream each member straight into pandas with encoding="utf-8-sig" rather than extracting it to disk. Those four decisions handle almost every archive-level surprise real feeds produce, and each of them costs a line. The tables inside are covered in understanding GTFS static feed structure.

Why an obviously present file cannot be opened Three archive-level surprises that make a member lookup fail on a perfectly valid feed. The file is in the archive but the open failed — why? nested in a folder Directory prefix resolve by basename, not by path named Stops.TXT Casing member names are case-sensitive opens, columns wrong Byte order mark read with utf-8-sig

Root Cause Analysis

A GTFS feed is a zip archive of CSV files, which sounds like it should be the least interesting part of an ingest. In practice it is where a surprising share of first-run failures happen, because the specification says what the files contain and very little about how they are packaged.

Directory prefixes. The specification expects the text files at the archive root. A meaningful minority of publishers zip a folder, so every member is gtfs/stops.txt or MBTA_20261102/stops.txt. Code that opens "stops.txt" raises KeyError on an entirely valid feed, and the error message names a file the operator can see plainly in the archive.

Casing. Stops.txt and STOPS.TXT both appear in the wild. Zip member names are case-sensitive, so an exact match fails on a file that is obviously present.

The byte order mark. Feeds exported from spreadsheet software routinely begin with a UTF-8 BOM. Read with plain utf-8, the mark becomes part of the first column’s name, so agency_id arrives as a string that looks identical in a log and matches nothing. This one is genuinely hard to diagnose from the symptom, because the printed column name looks right.

Size. stop_times.txt compresses about ten to one, so a 60 MB archive can expand to 600 MB, and a deliberately hostile one can expand to far more. Reading it into a naive parser allocates all of it before anything checks whether that was wise.

None of these is difficult. All of them are easier to handle once, in one loader, than to rediscover per pipeline.

Production-Ready Python Implementation

python 
"""Open a GTFS archive safely and stream its members."""
from __future__ import annotations

import logging
import posixpath
from dataclasses import dataclass
from pathlib import Path
from zipfile import BadZipFile, ZipFile

import pandas as pd

log = logging.getLogger("gtfs.archive")

# GTFS text compresses roughly 10:1. Far beyond that is a signal, not a feature.
MAX_COMPRESSION_RATIO = 200
MAX_UNCOMPRESSED_BYTES = 8 * 1024 ** 3          # 8 GiB across the whole archive
REQUIRED = ("agency.txt", "routes.txt", "trips.txt", "stop_times.txt", "stops.txt")


class ArchiveRejected(Exception):
    """The archive is unsafe or unusable — never a data-quality problem."""


@dataclass
class GtfsArchive:
    path: Path
    _zip: ZipFile
    _members: dict[str, str]          # lowercased basename -> real member name

    def __enter__(self) -> "GtfsArchive":
        return self

    def __exit__(self, *exc) -> None:
        self._zip.close()

    def has(self, name: str) -> bool:
        return name.lower() in self._members

    def open(self, name: str):
        try:
            return self._zip.open(self._members[name.lower()])
        except KeyError:
            raise FileNotFoundError(
                f"{self.path.name} has no member {name!r}; it contains "
                f"{sorted(self._members)}") from None

    def read_csv(self, name: str, **kwargs) -> pd.DataFrame:
        """Stream a member into pandas. Never extracts to disk."""
        with self.open(name) as handle:
            # utf-8-sig consumes a byte order mark if one is present, so the first
            # column keeps the name the rest of the pipeline expects.
            return pd.read_csv(handle, encoding="utf-8-sig",
                               keep_default_na=False, na_values=[""], **kwargs)

    @property
    def members(self) -> list[str]:
        return sorted(self._members)


def open_archive(path: Path) -> GtfsArchive:
    try:
        archive = ZipFile(path)
    except BadZipFile as exc:
        raise ArchiveRejected(f"{path.name} is not a readable zip archive: {exc}") from exc

    members: dict[str, str] = {}
    total_uncompressed = 0
    for info in archive.infolist():
        if info.is_dir():
            continue
        # Publishers routinely zip a FOLDER rather than its contents, so resolve
        # members by basename and let the prefix be whatever it likes.
        base = posixpath.basename(info.filename).lower()
        if not base.endswith(".txt"):
            continue
        if base in members:
            archive.close()
            raise ArchiveRejected(
                f"{path.name} contains {base} at two different paths "
                f"({members[base]} and {info.filename}) — which one is the feed?")
        members[base] = info.filename

        total_uncompressed += info.file_size
        if info.compress_size and info.file_size / info.compress_size > MAX_COMPRESSION_RATIO:
            archive.close()
            raise ArchiveRejected(
                f"{path.name}: {info.filename} declares a "
                f"{info.file_size / info.compress_size:.0f}:1 compression ratio "
                "— refusing to expand it")

    if total_uncompressed > MAX_UNCOMPRESSED_BYTES:
        archive.close()
        raise ArchiveRejected(
            f"{path.name} declares {total_uncompressed / 1024 ** 3:.1f} GiB "
            f"uncompressed, above the {MAX_UNCOMPRESSED_BYTES / 1024 ** 3:.0f} GiB limit")

    missing = [name for name in REQUIRED if name not in members]
    if missing:
        archive.close()
        raise ArchiveRejected(f"{path.name} is missing required member(s): {missing}")

    prefixes = {posixpath.dirname(v) for v in members.values()}
    if prefixes != {""}:
        log.info("%s: members are nested under %s — resolving by basename",
                 path.name, sorted(p for p in prefixes if p))

    log.info("%s: %d member(s), %.1f MB uncompressed",
             path.name, len(members), total_uncompressed / 1024 ** 2)
    return GtfsArchive(path=path, _zip=archive, _members=members)
What happens before a single row is parsed Build the member map, check the declared sizes from the archive index, confirm the required members exist, and only then stream anything. Map basename → member Size from the index, no reads Require the five core files Stream into pandas every check before the last one costs no decompression at all

Step-by-Step Walkthrough

The member map is built once, from basenames, lowercased. That single dictionary absorbs both the prefix problem and the casing problem, and it means every later lookup is exact and cheap. Callers ask for "stops.txt" and never learn that the file was actually MBTA_20261102/Stops.TXT.

A duplicate basename is refused rather than resolved. An archive containing stops.txt at two paths is genuinely ambiguous — often a nested copy of an older feed — and picking one silently means the pipeline may read a different file than an operator inspecting the archive would.

Sizes are checked from the central directory, before any read. ZipInfo.file_size and compress_size come from the archive’s own index, so the check costs no decompression at all. Both a per-member ratio and an archive-wide total are enforced, because a bomb can be one enormous member or a thousand moderate ones.

ArchiveRejected is a distinct exception. Everything it signals is a packaging or safety problem, categorically different from a feed whose data is wrong. A batch runner can retry a network failure, quarantine a data-quality failure, and page someone about an ArchiveRejected — which it cannot do if all three arrive as ValueError.

read_csv passes keep_default_na=False with an explicit na_values. Without it, pandas treats the literal strings NA, null and None as missing — and NA is a real stop_id in more than one feed. Only an empty field should be missing.

open returns the zip’s own file object. pandas reads it as a stream, so a 600 MB stop_times.txt is parsed without ever existing as a file. That matters for both disk and for the memory-efficient reading strategies that build on top of it.

Verification and Output

python 
def verify(archive: GtfsArchive) -> None:
    for name in REQUIRED:
        assert archive.has(name), f"required member {name} vanished after open"

    agency = archive.read_csv("agency.txt", dtype="string")
    assert not agency.empty, "agency.txt is present but empty"
    assert not any(c.startswith("") for c in agency.columns), (
        "a byte order mark survived into a column name — check the encoding")
    assert "agency_timezone" in agency.columns, (
        "agency.txt has no agency_timezone; every time in this feed is meaningless")

    head = archive.read_csv("stop_times.txt", dtype="string", nrows=5)
    assert {"trip_id", "stop_id", "stop_sequence"} <= set(head.columns), (
        f"stop_times.txt is missing key columns; it has {list(head.columns)}")

Reading nrows=5 from stop_times.txt verifies the largest member is parseable without paying to parse it, which is exactly the check to run before committing to a full ingest.

A well-formed archive:

text 
INFO gtfs.archive: mbta_20261102.zip: 14 member(s), 612.4 MB uncompressed

A nested one, handled rather than failed:

text 
INFO gtfs.archive: regional_20261102.zip: members are nested under ['gtfs_feed'] — resolving by basename
INFO gtfs.archive: regional_20261102.zip: 9 member(s), 84.1 MB uncompressed

And the refusals:

text 
ArchiveRejected: bad_feed.zip: gtfs/stop_times.txt declares a 4108:1 compression ratio — refusing to expand it
ArchiveRejected: partial.zip is missing required member(s): ['stop_times.txt', 'stops.txt']
Compression ratios across GTFS members How far each member expands: the text tables compress about ten to one, which is what makes a far higher declared ratio a signal rather than a feature. stop_times.txt 11:1 highly repetitive text shapes.txt 9:1 repeated coordinate prefixes trips.txt 8:1 stops.txt 5:1 more varied text A refused archive 4108:1 declared, not measured the ratio comes from the archive index, so checking it is free

Gotchas and Edge Cases

  • Zip64 archives. Python’s zipfile handles them transparently, but some older tooling in a pipeline may not. If an archive exceeds 4 GiB uncompressed, check anything downstream that touches the raw file.
  • Encodings other than UTF-8. The specification requires UTF-8, and a few feeds ship Latin-1 anyway. utf-8-sig will raise on those; catching UnicodeDecodeError and retrying with cp1252 is a pragmatic fallback, but log it loudly — the feed is out of specification and stop names will be subtly wrong.
  • Members with a leading ./ or backslashes. Windows-built archives sometimes use backslashes as separators, which posixpath.basename will not split. Normalising with filename.replace("\\", "/") before taking the basename covers it.
  • Empty optional members. A zero-byte calendar.txt is different from an absent one, and both are legal. has() reports presence; whether the content is usable is a question for the calendar reader.
  • Reading the same member twice. Each open returns a fresh stream, so this is safe — but it decompresses twice. For anything read more than once, read it into a frame and keep the frame.
  • Archives fetched over the network into memory. ZipFile accepts any seekable file-like object, so a BytesIO works unchanged. Do not accept a non-seekable stream: the central directory lives at the end of the file, so the size checks would be impossible.

Frequently Asked Questions

Why does my first column name have strange characters in it?

The file starts with a UTF-8 byte order mark, which pandas reads as part of the first column’s name — so agency_id becomes a name that no lookup matches. Read with encoding utf-8-sig and the mark is consumed correctly.

Is it safe to assume the files sit at the root of the archive?

No. A fair number of publishers zip a folder rather than its contents, so every member is prefixed with something like gtfs/ or the feed date. Resolve members by their basename rather than by an exact path.

Should I extract the archive to disk first?

Not usually. ZipFile.open gives a file object pandas can read directly, which avoids writing a gigabyte of CSV to disk only to read it straight back. Extract only when a tool you do not control demands a real path.

What is a decompression bomb and can a GTFS feed be one?

An archive whose compressed size is trivial and whose uncompressed size is enormous. GTFS text compresses roughly ten to one, so a ratio far beyond that is a signal to stop rather than to allocate. Checking the declared sizes before reading costs nothing.