Matching Realtime to Static Schedules

A GTFS-Realtime feed is deliberately thin. A TripUpdate might say “trip 4471 is 90 seconds late at stop 218” and nothing more — no route name, no headsign, no scheduled time, no stop name. All of that context lives in the static feed, and producing anything a rider can read means joining the two together correctly. That join looks trivial until you meet a trip_id that exists in the realtime feed but not in trips.txt, a trip that ran yesterday reusing today’s identifier, or an extra trip the dispatcher added an hour ago that no static file has ever seen. This guide, part of the GTFS-Realtime Integration overview, works through the join keys between realtime and static GTFS and the reconciliation logic that keeps the match honest.

The Reconciliation Problem

The static and realtime halves of GTFS are published on different cadences by different systems. The static feed is a versioned archive refreshed every few weeks; the realtime feed is a live stream refreshed every few seconds. Nothing in the specification forces the two to agree on identifiers at any given instant, and in practice they drift constantly. A scheduling department regenerates trip_id values during a booking rebuild, but the realtime system keeps emitting the previous run’s IDs until its next sync. An operations desk inserts an extra late-night trip that exists only in the realtime stream. A trip is cancelled, and the realtime feed flags it while trips.txt still lists it.

Each of these is normal, specified behaviour — not corruption — and a robust consumer must expect all of them. Treating a failed trip_id join as an error produces alarming logs and dropped predictions; treating it as routine, and falling back through a hierarchy of weaker keys, produces a stable departure board. The workflow below is production-grade whether you are matching one agency’s feed or a merged multi-agency stream.

Prerequisites

Before running the code in this guide, confirm the following:

• Python 3.9+ with pandas and the realtime bindings installed:

pip install gtfs-realtime-bindings pandas

• A decoded realtime feed. This guide starts from FeedEntity objects; if you cannot yet produce them, begin with decoding GTFS-Realtime protobuf feeds.
• A parsed static feed. The join targets — trips.txt, routes.txt, stop_times.txt, and calendar.txt/calendar_dates.txt — should already be loaded; the pandas and partridge parsing guide covers strict-typed loading, and understanding GTFS static feed structure covers how those files relate.
• The feed’s service calendar logic, so you can resolve which service_id values are active on a given start_date.

Concept and Spec Background

The join keys

Every realtime entity that references the schedule does so through a TripDescriptor, and stop-level entities add a StopTimeUpdate. These carry the only identifiers you can join on:

TripDescriptor and schedule_relationship

The TripDescriptor.schedule_relationship enum is the single most important field for a correct join, because it tells you how the trip relates to the schedule before you attempt any match:

The default is SCHEDULED, so a producer that omits the field is asserting the trip is on the timetable. Read this field first and branch on it; do not attempt a blind trip_id merge and then puzzle over the misses.

Why trip_id drifts

A trip_id is only guaranteed unique and stable within a single static feed version. When an agency republishes the static feed — a routine event — it may regenerate trip_id values wholesale from its scheduling system. The realtime feed, sourced from an operations system on its own refresh cycle, can lag that change by minutes to hours. During the overlap, realtime trip_id values reference a schedule version you no longer hold. This is not an error to be fixed; it is a synchronisation window to be tolerated, which is why the fallback hierarchy below never relies on trip_id alone.

Service-date scoping

A trip_id identifies a pattern, not a run. The 08:15 outbound exists as one trip_id that operates every weekday, so the same trip_id recurs across many calendar days. The realtime start_date (YYYYMMDD) pins the run to one service day. Joining without carrying start_date risks attributing a delay to the wrong day around midnight, when yesterday’s late-running trip and today’s early departure share the identifier.

Step-by-Step Implementation

Step 1: Extract TripDescriptor and StopTimeUpdate Keys

Flatten each realtime entity into two frames — one trip-level, one stop-level — carrying the schedule_relationship so later steps can branch on it.

import logging
import pandas as pd
from google.transit import gtfs_realtime_pb2

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

TripDescriptor = gtfs_realtime_pb2.TripDescriptor
REL_NAME = TripDescriptor.ScheduleRelationship.Name


def extract_rt_keys(feed: gtfs_realtime_pb2.FeedMessage) -> tuple[pd.DataFrame, pd.DataFrame]:
    trip_rows, stop_rows = [], []
    for entity in feed.entity:
        if not entity.HasField("trip_update"):
            continue
        trip = entity.trip_update.trip
        rel = REL_NAME(trip.schedule_relationship)
        trip_rows.append(
            {
                "entity_id": entity.id,
                "trip_id": trip.trip_id or None,
                "route_id": trip.route_id or None,
                "start_date": trip.start_date or None,   # 'YYYYMMDD'
                "start_time": trip.start_time or None,
                "schedule_relationship": rel,
            }
        )
        for stu in entity.trip_update.stop_time_update:
            stop_rows.append(
                {
                    "trip_id": trip.trip_id or None,
                    "start_date": trip.start_date or None,
                    "stop_sequence": stu.stop_sequence if stu.HasField("stop_sequence") else pd.NA,
                    "stop_id": stu.stop_id or None,
                }
            )
    return pd.DataFrame(trip_rows), pd.DataFrame(stop_rows)

start_date is kept as the raw YYYYMMDD string; it is a calendar key, not a timestamp, and parsing it to a datetime here would only invite timezone confusion.

Step 2: Scope the Static Feed to the Service Date

Resolve which service_id values run on the realtime start_date, then restrict trips.txt to those services. This prevents matching a trip_id pattern to a day it does not operate.

def active_service_ids(calendar: pd.DataFrame, calendar_dates: pd.DataFrame, yyyymmdd: str) -> set:
    """Service IDs running on a given YYYYMMDD, honouring calendar_dates exceptions."""
    day = pd.to_datetime(yyyymmdd, format="%Y%m%d")
    weekday_col = day.day_name().lower()   # e.g. 'tuesday'

    base = calendar[
        (calendar["start_date"] <= int(yyyymmdd))
        & (calendar["end_date"] >= int(yyyymmdd))
        & (calendar[weekday_col] == 1)
    ]
    active = set(base["service_id"])

    exceptions = calendar_dates[calendar_dates["date"] == int(yyyymmdd)]
    active |= set(exceptions.loc[exceptions["exception_type"] == 1, "service_id"])  # added
    active -= set(exceptions.loc[exceptions["exception_type"] == 2, "service_id"])  # removed
    return active

The calendar_dates.txt exceptions must be applied after the base calendar, because exception_type == 2 can remove a service that the weekly pattern would otherwise include (a holiday, for instance).

Step 3: Join on trip_id, Then Fall Back

Branch on schedule_relationship, join SCHEDULED trips on trip_id scoped to active services, and route the misses into a route_id + start_time fallback.

trips_static = pd.read_csv(
    "gtfs/trips.txt",
    dtype={"route_id": str, "service_id": str, "trip_id": str, "trip_headsign": str, "direction_id": "Int64"},
)

rt_trips, rt_stops = extract_rt_keys(feed)
service_ids = active_service_ids(calendar, calendar_dates, rt_trips["start_date"].dropna().iloc[0])
trips_today = trips_static[trips_static["service_id"].isin(service_ids)]

scheduled = rt_trips[rt_trips["schedule_relationship"].isin(["SCHEDULED", "CANCELED"])]
matched = scheduled.merge(
    trips_today[["trip_id", "route_id", "trip_headsign", "direction_id"]],
    on="trip_id",
    how="left",
    suffixes=("_rt", "_static"),
    indicator=True,
)

unmatched = matched[matched["_merge"] == "left_only"]
logging.info("trip_id join: %d matched, %d unmatched", (matched["_merge"] == "both").sum(), len(unmatched))

Keeping indicator=True turns the fallback decision into a data filter rather than a guess: every left_only row is a realtime trip whose trip_id did not resolve against today’s schedule and must be reconciled by other means.

Step 4: Handle ADDED, CANCELED, and Drifted Trips

ADDED trips are built entirely from realtime data; CANCELED trips match but suppress predictions; drifted SCHEDULED misses fall back to route_id + start_time.

added = rt_trips[rt_trips["schedule_relationship"] == "ADDED"].copy()
added["source"] = "realtime_only"      # no trips.txt row exists — expected

# Drifted SCHEDULED misses: recover via route_id + first scheduled departure
first_dep = (
    pd.read_csv("gtfs/stop_times.txt", dtype={"trip_id": str, "departure_time": str, "stop_sequence": "Int64"})
    .sort_values(["trip_id", "stop_sequence"])
    .groupby("trip_id", as_index=False)
    .first()[["trip_id", "departure_time"]]
    .rename(columns={"departure_time": "start_time"})
)
drift_candidates = unmatched.merge(
    trips_today.merge(first_dep, on="trip_id"),
    on=["route_id", "start_time"],
    how="inner",
    suffixes=("_rt", "_recovered"),
)
logging.info("Recovered %d drifted trips via route_id + start_time", len(drift_candidates))

An ADDED trip returning zero trips.txt rows is correct, not a failure — flag its provenance so the display layer knows the headsign and route name came from the realtime stream, not the schedule.

Step 5: Join Stops on (trip_id, stop_sequence)

Attach each StopTimeUpdate to its scheduled stop. Prefer stop_sequence; fall back to stop_id only where the producer omitted the sequence.

stop_times = pd.read_csv(
    "gtfs/stop_times.txt",
    dtype={"trip_id": str, "stop_id": str, "stop_sequence": "Int64", "arrival_time": str, "departure_time": str},
)

has_seq = rt_stops[rt_stops["stop_sequence"].notna()]
joined_seq = has_seq.merge(stop_times, on=["trip_id", "stop_sequence"], how="left", suffixes=("_rt", ""))

no_seq = rt_stops[rt_stops["stop_sequence"].isna()]
joined_stop = no_seq.merge(stop_times, on=["trip_id", "stop_id"], how="left", suffixes=("_rt", ""))

Joining on stop_id alone is unsafe on trips that visit a stop twice — a loop route or an out-and-back branch — which is why stop_sequence is the primary key. The detailed mechanics of this stop-level join are the subject of joining TripUpdates to stop_times.txt.

Validation and Verification

Gate the pipeline on coverage and key integrity before publishing predictions:

def validate_match(rt_trips: pd.DataFrame, matched: pd.DataFrame) -> dict:
    """Assert join invariants; return coverage metrics."""
    scheduled_ct = (rt_trips["schedule_relationship"] == "SCHEDULED").sum()
    matched_ct = (matched["_merge"] == "both").sum()
    coverage = matched_ct / scheduled_ct if scheduled_ct else 1.0

    # Any SCHEDULED trip missing start_date cannot be safely service-scoped
    missing_date = rt_trips[(rt_trips["schedule_relationship"] == "SCHEDULED") & rt_trips["start_date"].isna()]
    assert missing_date.empty, f"{len(missing_date)} SCHEDULED trips lack start_date"

    # Coverage below 80% usually signals a stale static feed version
    assert coverage >= 0.80, f"trip_id match coverage {coverage:.0%} — static feed may be out of date"
    return {"scheduled": int(scheduled_ct), "matched": int(matched_ct), "coverage": round(coverage, 3)}

Coverage is the single most useful health metric. A sudden drop from near-100% to 60% almost always means the static feed was republished with regenerated trip_id values and your consumer has not yet reloaded it — exactly the drift scenario Step 4 exists to absorb.

Failure Modes and Edge Cases

• Static feed version skew. The realtime feed references a schedule version you have already replaced. Keep the two or three most recent static versions loaded and match against the version whose validity window contains start_date, rather than only the newest. Feed-versioning discipline is covered in agency metadata and feed versioning practices.

• Midnight and times past 24:00. GTFS times can exceed 24:00:00 for trips that cross midnight, and start_date refers to the service day, not the calendar day. A trip departing at 25:10:00 on start_date=20260713 actually runs at 01:10 on the 14th. Resolve this the same way timezone normalization handles overnight service before comparing to wall-clock time.

• Frequency-based (UNSCHEDULED) trips. These have no unique trip_id per run; match them on route_id + start_time after expanding the schedule as described for frequency-based schedules. A blind trip_id join will either miss or collide.

• Reused stop_id on loop routes. As noted, a stop_id join fans out on trips that touch a stop twice. Always prefer stop_sequence; only fall back to stop_id when the producer omits it, and log when you do.

• ADDED trips with partial TripDescriptor. Some producers emit an ADDED trip with a trip_id that coincidentally collides with a scheduled one. Because the schedule_relationship says ADDED, honour that field over the coincidental match — never let a trip_id collision override an explicit relationship flag. Catching this kind of contradiction is what GTFS validation rules formalise.

Performance and Scale Notes

The join itself is cheap; the cost is re-reading and re-typing the static feed on every poll. A realtime feed refreshes every 10–30 seconds, but trips.txt and stop_times.txt change only when a new static version is published. Load and index them once.

# Load static tables once at startup, index for repeated realtime joins
trips_static = pd.read_csv(
    "gtfs/trips.txt",
    dtype={"route_id": str, "service_id": str, "trip_id": str},
).set_index("trip_id")

stop_times_static = (
    pd.read_csv(
        "gtfs/stop_times.txt",
        dtype={"trip_id": str, "stop_id": str, "stop_sequence": "Int64",
               "arrival_time": str, "departure_time": str},
    )
    .set_index(["trip_id", "stop_sequence"])
    .sort_index()
)

# Each poll now joins against the cached, pre-indexed frames rather than re-reading CSV.

For national feeds where stop_times.txt runs to tens of millions of rows, persist the indexed static tables to Parquet and memory-map them, applying the strategies in memory-efficient processing for large feeds. A merged multi-agency stream should prefix identifiers with agency_id before any join, because trip_id and stop_id collide across agencies almost by default.

Frequently Asked Questions

Related

• Joining TripUpdates to stop_times.txt — the complete stop-level merge that attaches live delays to every scheduled row
• Handling GTFS Service Alerts — the same selector-to-static join applied to alert informed_entity[] scoping
• Processing TripUpdates and Delays — where the matched trips become arrival predictions
• Mastering stops.txt and stop_times.txt Relationships — the composite-key model that the stop-level join depends on
• Parsing GTFS with pandas and partridge — strict-typed loading of the static tables you join against

Up: GTFS-Realtime Integration | Home