Merging Multiple GTFS Feeds in Python

Before you concatenate feeds from different agencies, prefix every identifier column — stop_id, trip_id, route_id, service_id, shape_id, and the foreign keys that reference them — with a per-feed namespace such as the agency_id. Because GTFS only guarantees identifiers are unique within one feed, two operators will happily both use stop_id values of 1, 2, 3; namespacing every key column in the same pass keeps each agency’s foreign-key references intact while making collisions impossible after the tables are stacked.

Root Cause Analysis

The GTFS specification scopes identifier uniqueness to a single feed. Inside one agency’s export, route_id RED is unambiguous. The moment you place two agencies’ files side by side, that guarantee evaporates: a metro operator and a bus operator may both label a route RED, both number their first stop 1, and both name a weekday service WKDY. Nothing in either feed is wrong — the identifiers were simply never meant to be globally unique.

Naive concatenation turns this latent ambiguity into silent data corruption. When you concat two stops.txt files and then join the combined stop_times.txt back on stop_id, a row that meant “stop 1 on the metro” now also matches “stop 1 on the bus network”. The join fans out, trip geometries cross the city, and passenger-facing tools show departures from the wrong platform. Nothing raises an exception, because every stop_id still resolves to a row — just the wrong one, or several.

The structural fix is namespacing: rewrite each identifier so it carries its source agency in the value itself. If the metro’s stops become metro:1, metro:2 and the bus’s become bus:1, bus:2, they can never collide, and a stop_times.txt row that referenced 1 under the metro feed must be rewritten to metro:1 in the very same operation so the reference still points where it always did. The invariant to hold onto is that a key and every reference to that key must be transformed by exactly the same rule. This is the same discipline that underpins the stops.txt and stop_times.txt relationship model: a composite of primary keys and the foreign keys that point at them, which must move together or not at all.

Production-Ready Python Implementation

The script below merges an arbitrary number of GTFS directories. For each feed it derives a namespace, then rewrites every identifier column — both the columns that define keys and the columns that reference them — before concatenating table by table with pandas. A key map declares which columns are identifiers per file so the same prefix rule is applied everywhere a value appears.

import logging
from pathlib import Path

import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

# For each GTFS file, list the columns that are primary or foreign keys.
# Every listed column is prefixed with the feed's namespace so that a key
# and all references to it move together.
ID_COLUMNS = {
    "agency.txt": ["agency_id"],
    "stops.txt": ["stop_id", "parent_station"],
    "routes.txt": ["route_id", "agency_id"],
    "trips.txt": ["route_id", "service_id", "trip_id", "shape_id", "block_id"],
    "stop_times.txt": ["trip_id", "stop_id"],
    "calendar.txt": ["service_id"],
    "calendar_dates.txt": ["service_id"],
    "shapes.txt": ["shape_id"],
    "transfers.txt": ["from_stop_id", "to_stop_id"],
    "frequencies.txt": ["trip_id"],
}

# Keep every id column as a string so numeric-looking ids are not coerced.
STRING_DTYPE = str


def _namespace(value: object, prefix: str) -> object:
    """Prefix a single id value, leaving blanks (optional FKs) untouched."""
    if pd.isna(value) or value == "":
        return value
    return f"{prefix}:{value}"


def load_and_namespace(feed_dir: Path, prefix: str) -> dict[str, pd.DataFrame]:
    """Read one feed and rewrite every id column with the given namespace."""
    tables: dict[str, pd.DataFrame] = {}
    for filename, key_cols in ID_COLUMNS.items():
        path = feed_dir / filename
        if not path.exists():
            continue  # optional files (transfers, frequencies) may be absent
        dtypes = {col: STRING_DTYPE for col in key_cols}
        table = pd.read_csv(path, dtype=dtypes)
        for col in key_cols:
            if col in table.columns:
                table[col] = table[col].map(lambda v: _namespace(v, prefix))
        tables[filename] = table
        logging.info("%-18s | %-8s | %6d rows", filename, prefix, len(table))
    return tables


def merge_feeds(feeds: dict[str, str], out_dir: str) -> None:
    """
    Merge multiple GTFS feeds into one directory.

    Parameters
    ----------
    feeds : dict[str, str]
        Mapping of namespace prefix -> path to that agency's GTFS directory.
    out_dir : str
        Directory to write the combined feed into.
    """
    if len(set(feeds.keys())) != len(feeds):
        raise ValueError("Namespace prefixes must be unique across feeds")

    collected: dict[str, list[pd.DataFrame]] = {}
    for prefix, feed_path in feeds.items():
        tables = load_and_namespace(Path(feed_path), prefix)
        for filename, table in tables.items():
            collected.setdefault(filename, []).append(table)

    out_path = Path(out_dir)
    out_path.mkdir(parents=True, exist_ok=True)

    for filename, frames in collected.items():
        combined = pd.concat(frames, ignore_index=True, sort=False)
        combined.to_csv(out_path / filename, index=False)
        logging.info("Wrote %-18s | %6d combined rows", filename, len(combined))

    logging.info("Merged %d feeds into %s", len(feeds), out_dir)


if __name__ == "__main__":
    merge_feeds(
        feeds={
            "metro": "/data/gtfs/metro",
            "bus": "/data/gtfs/citybus",
            "rail": "/data/gtfs/regional_rail",
        },
        out_dir="/data/gtfs/combined",
    )

Step-by-Step Walkthrough

A single source of truth for key columns. ID_COLUMNS declares, per file, exactly which columns are identifiers. This is the crux of the whole approach: because stop_id is listed under both stops.txt (where it is defined) and stop_times.txt (where it is referenced), the same prefix rule rewrites both, and the foreign key still resolves after the merge.

Namespacing handles optional foreign keys. _namespace returns blanks unchanged. Columns like parent_station, shape_id, and block_id are frequently empty; prefixing an empty string would fabricate a reference to a non-existent metro: entity. Leaving blanks alone preserves the “no parent” and “no shape” semantics.

Every id column is read as a string. The dtypes map pins each key column to str at read time. Agencies that use purely numeric identifiers would otherwise have them loaded as integers, and 12 prefixed as metro:12 in one table but referenced as integer 12 in another would fail to join.

Concatenation is grouped by filename. collected accumulates one list of frames per GTFS file across all feeds, then a single pd.concat stacks them. Using sort=False preserves column order, and ignore_index=True produces a clean combined row index. Optional files that only some feeds provide are concatenated across just the feeds that have them.

Prefix uniqueness is enforced up front. The guard at the top of merge_feeds rejects duplicate namespaces, since two feeds sharing the prefix metro would reintroduce exactly the collisions the approach exists to prevent.

Verification and Output

After merging, prove that no identifier collides and that every foreign key still resolves in the combined feed.

import pandas as pd
from pathlib import Path

combined = Path("/data/gtfs/combined")

stops = pd.read_csv(combined / "stops.txt", dtype={"stop_id": str})
trips = pd.read_csv(combined / "trips.txt", dtype={"trip_id": str, "route_id": str, "service_id": str})
stop_times = pd.read_csv(combined / "stop_times.txt", dtype={"trip_id": str, "stop_id": str})

# 1. No duplicate primary keys after the merge
assert stops["stop_id"].is_unique, "Duplicate stop_id in merged feed"
assert trips["trip_id"].is_unique, "Duplicate trip_id in merged feed"

# 2. Every stop_times foreign key resolves
orphan_stops = set(stop_times["stop_id"]) - set(stops["stop_id"])
orphan_trips = set(stop_times["trip_id"]) - set(trips["trip_id"])
assert not orphan_stops, f"stop_times references unknown stop_id: {list(orphan_stops)[:5]}"
assert not orphan_trips, f"stop_times references unknown trip_id: {list(orphan_trips)[:5]}"

# 3. Every id carries a namespace prefix
assert stops["stop_id"].str.contains(":").all(), "Un-namespaced stop_id present"

print(f"Merged feed OK: {len(stops)} stops, {len(trips)} trips, {len(stop_times)} stop_times")

A clean merge prints stop, trip, and stop_times counts equal to the sum of the source feeds, with all three assertions passing. If orphan_trips is non-empty, a trip_id was prefixed in stop_times.txt but not in trips.txt (or vice versa) — usually because a key column was omitted from ID_COLUMNS. Adding it there and re-running fixes the reference in a single pass.

Gotchas and Edge Cases

• transfers.txt and other cross-reference files. Transfers reference stops via from_stop_id and to_stop_id; both must be namespaced. If you intend to build cross-agency transfers between, say, a rail station and an adjacent bus stop, you must add those transfer rows explicitly after the merge — they cannot exist in either source feed because neither feed knows about the other’s stops.

• agency.txt with a blank agency_id. Single-agency feeds may omit agency_id entirely, and routes.txt may then also omit it. When merging, synthesize an agency_id from your namespace so every route is attributable; otherwise routes from different operators become indistinguishable in the combined routes.txt.

• Service calendars that overlap by date. Two agencies can both define a service_id active on the same dates. Namespacing keeps them distinct, but be aware the merged feed now has more service_id values than any single feed — downstream date-filtering (for example with the loader described in parsing GTFS with pandas and partridge) must account for all of them.

• Feed size after merging. A combined national feed can be several times larger than any input. If stop_times.txt grows past what fits comfortably in memory, stream the concatenation to disk per file rather than holding every frame at once; the memory-efficient processing guide covers chunked writes and columnar caching for exactly this case.

Frequently Asked Questions

Related

• Detecting Duplicate Trips in GTFS Feeds — a useful follow-up when merged feeds introduce overlapping service on shared corridors
• Fixing Missing stop_times.txt Records in Python — the referential-integrity checks that catch a mis-prefixed key
• Memory-Efficient Processing for Large Feeds — keep a multi-agency stop_times.txt within RAM after the merge
• Up: Batch Processing Strategies for Multi-Agency Feeds — the wider batch and orchestration patterns this merge fits into
• Section: Python Parsing & Data Normalization — the full GTFS parsing and normalization pipeline · Home