GTFS-Realtime Integration

The static General Transit Feed Specification tells you when a bus is supposed to arrive. GTFS-Realtime tells you where it actually is, how late it is running, and whether the line is suspended altogether. Riders judge a transit app almost entirely on the accuracy of that live layer, and yet it is the part most engineering teams get wrong — because the real-time feed is a binary protobuf stream with its own timestamp conventions, its own entity model, and a fragile dependency on a static feed that may have been republished underneath it minutes ago. A single unmatched trip_id, a decoded FeedMessage from the wrong protobuf version, or a POSIX timestamp interpreted as a local clock string produces confidently wrong ETAs that erode rider trust faster than showing nothing at all.

This guide covers the complete engineering picture of the live layer: how to poll and decode the protobuf FeedMessage, how the three entity types (VehiclePositions, TripUpdates, ServiceAlerts) are structured, how each references the static feed structure you already ingest, and how to fuse the two into stop-level arrival predictions in Python. It builds directly on the static-side patterns — treat this as the moving picture drawn on top of the schedule baseline.

Real-Time Pipeline Overview

A GTFS-Realtime consumer runs a tight loop that never stops: fetch bytes, decode them into a structured message, fan the entities out by type, reconcile each against the static schedule, and serve the merged view. Unlike static ingestion, which runs on a daily or weekly cadence, this loop typically cycles every 10–30 seconds and must degrade gracefully when the upstream endpoint stalls or returns a truncated body.

The critical architectural point is that the middle stage — decode — produces one flat FeedMessage.entity list, and every downstream stage depends on correctly typing and routing those entities. A consumer that treats the stream as opaque JSON, or that re-fetches the static feed on every cycle, will not keep up with the polling cadence. The static schedule is loaded once, indexed by trip_id, and held in memory; the real-time loop joins against that resident index on every tick.

Core Concepts: The Three Entity Types

Every GTFS-Realtime FeedMessage contains a header (with a gtfs_realtime_version, an incrementality flag, and a feed timestamp) followed by a repeated list of FeedEntity records. Each FeedEntity is a tagged union: it carries exactly one of a vehicle (VehiclePosition), a trip_update (TripUpdate), or an alert (Alert), plus a string id that is unique only within that message. Understanding which fields each entity exposes — and precisely how each one points back into the static feed — is the whole game.

Three details trip up almost every first implementation. First, protobuf fields are optional by construction — a missing bearing or delay is not zero, it is absent, and you must call HasField(...) rather than reading a default-zero value that looks like real data. Second, every timestamp is a POSIX epoch integer in UTC seconds, a fundamentally different representation from the noon-relative local clock strings in stop_times.txt; reconciling the two requires the same timezone normalization discipline the static side demands, applied in reverse. Third, the TripDescriptor shared by all three entity types can identify a trip either by an explicit trip_id or, for frequency-based service, by a (route_id, start_time, start_date) tuple — so a robust matcher handles both addressing modes.

Decoding the Protobuf Feed

Everything downstream depends on turning the raw response body into a typed FeedMessage. The canonical approach uses the gtfs-realtime-bindings package, which ships the pre-compiled gtfs_realtime_pb2 module so you never run protoc yourself. The decode itself is two lines — instantiate gtfs_realtime_pb2.FeedMessage() and call ParseFromString(response.content) — but the surrounding concerns (content-type validation, handling a truncated body, distinguishing an HTML error page from a real feed) are where production robustness lives.

The main tradeoff at this stage is between decoding into the native protobuf object graph versus flattening immediately into a pandas DataFrame. The protobuf objects are memory-light and preserve the HasField semantics that let you distinguish absent from zero, but they are awkward to join and aggregate. Flattening to a DataFrame makes the Python parsing and normalization toolkit available for matching and analytics, at the cost of having to encode presence explicitly (e.g. NaN for absent delay) so you do not lose the absent-versus-zero distinction. Most pipelines decode to protobuf, apply per-entity presence checks, then emit a typed DataFrame per entity class.

Polling cadence is the other half of this topic: poll too slowly and positions go stale between ticks; poll too aggressively and agencies rate-limit or ban your client. Respect Cache-Control and ETag response headers, back off on 429 and 5xx, and align your interval to the feed’s own header.timestamp refresh rate rather than a fixed guess. These decode and polling mechanics — including conditional requests and truncated-body detection — are worked end to end in decoding GTFS-Realtime protobuf feeds.

import logging
import requests
from google.transit import gtfs_realtime_pb2

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("gtfsrt.decode")

def fetch_feed_message(endpoint_url: str, timeout: float = 10.0) -> gtfs_realtime_pb2.FeedMessage:
    response = requests.get(endpoint_url, timeout=timeout)
    response.raise_for_status()
    feed_message = gtfs_realtime_pb2.FeedMessage()
    feed_message.ParseFromString(response.content)
    logger.info(
        "decoded FeedMessage version=%s entities=%d feed_ts=%s",
        feed_message.header.gtfs_realtime_version,
        len(feed_message.entity),
        feed_message.header.timestamp,
    )
    return feed_message

Tracking Vehicle Positions

A VehiclePosition answers “where is it now” with a position sub-message carrying latitude, longitude, and — when the agency provides it — bearing and speed. The current_stop_sequence and current_status fields locate the vehicle relative to its scheduled stops (INCOMING_AT, STOPPED_AT, IN_TRANSIT_TO), which is often more actionable for a rider-facing map than raw coordinates. Because coordinates are WGS84 decimal degrees — identical to the convention in the static stops.txt — you can compute distance-to-next-stop directly against the static feed’s stop geometry without reprojection for the display case, though metric distance and speed calculations still want a projected CRS.

The recurring tradeoff is trusting the reported bearing/speed versus deriving them yourself. Many agencies leave those fields absent or emit stale GPS-derived values; deriving bearing and speed from successive position samples of the same vehicle.id is often more reliable, but requires you to retain the previous tick’s position and guard against duplicate or out-of-order timestamps. The other subtlety is that a vehicle block can serve multiple trips in sequence, so vehicle.trip.trip_id changes mid-stream even though vehicle.id stays constant — keyed state must be indexed by vehicle.id, not trip_id. Position handling, speed/bearing derivation, and mapping vehicles to their scheduled stop sequence are covered in tracking vehicle positions in realtime.

Processing TripUpdates and Delays

The TripUpdate is the workhorse of arrival prediction. Each one carries a TripDescriptor and a repeated stop_time_update list, where every entry pins a stop_id (or stop_sequence) to an arrival and/or departure StopTimeEvent. That event exposes two ways to express timing: an absolute time (POSIX seconds) or a relative delay (signed seconds against the scheduled time). Crucially, GTFS-Realtime allows an agency to send an update for only some stops and have the delay propagate forward to all later stops until the next explicit update — so you cannot assume every upcoming stop has its own stop_time_update row.

This propagation rule is the main source of subtle bugs. The correct algorithm joins each trip’s stop_time_update entries onto its full ordered stop list from stop_times.txt, then forward-fills the delay along stop_sequence so unlisted stops inherit the last known delay. Converting a delay into a wall-clock ETA means adding it to the scheduled time — which requires the static arrival_time for that (trip_id, stop_sequence), another reason the stop_times and stops relationship is load-bearing here. The tradeoff between using the absolute time versus the relative delay matters when the two disagree: prefer time when present because it already encodes the agency’s own schedule assumptions, and fall back to scheduled_time + delay otherwise. Prediction computation and the forward-fill mechanics are detailed in processing TripUpdates and delays.

A second complication is the schedule_relationship enum on each stop_time_update, which can mark a stop as SKIPPED, NO_DATA, or ADDED. A SKIPPED stop must be removed from the predicted itinerary entirely rather than shown with an inherited delay, and NO_DATA means the agency explicitly declines to predict — showing the static scheduled time is more honest than forward-filling a stale delay across it.

Handling Service Alerts

Alerts are the qualitative layer: free-text, human-readable disruption notices scoped to specific parts of the network. An Alert couples one or more active_period windows (POSIX start/end ranges) with a list of informed_entity selectors, each of which can target a route_id, a stop_id, an agency_id, or a specific trip. The cause and effect enums (CONSTRUCTION, ACCIDENT, DETOUR, NO_SERVICE, SIGNIFICANT_DELAYS) drive iconography, while header_text and description_text are TranslatedString bundles that may carry the same message in several languages.

The design tension in alert processing is scope resolution. A single alert with informed_entity naming only a route_id implicitly affects every stop and trip on that route, so a stop-level UI must expand route-scoped alerts down to the stops that route serves — a join back through trips.txt and stop_times.txt. Over-expansion floods every stop with banners; under-expansion hides a relevant disruption from a rider standing at an affected platform. The active_period filtering is equally easy to get wrong: an alert present in the feed is not necessarily active now, and an empty active_period means “active whenever it appears in the feed.” Selecting the right TranslatedString by language and correctly expanding informed_entity scope are handled in handling GTFS service alerts.

Matching Realtime to Static Schedules

None of the three entity types is useful in isolation — their value comes entirely from being fused with the static schedule. This is the reconciliation stage, and it is where most integration effort actually goes. The join key is trip_id, resolved against the static trips.txt, then extended to the stop level by joining stop_time_update.stop_id (or current_stop_sequence) against the ordered rows of stop_times.txt. The output is an enriched record per trip that knows its route, its headsign, its full ordered stop list, its scheduled times, and its live delay — everything a prediction endpoint needs.

The hard part is that the two feeds drift. A real-time feed references a trip_id that no longer exists because the agency republished the static feed with regenerated identifiers; or it uses the frequency-based (route_id, start_time, start_date) addressing mode that has no literal trip_id at all. Matching must therefore be layered: attempt a direct trip_id join first, fall back to route-plus-start-time resolution, and quarantine whatever still fails to match rather than dropping it silently — an unmatched-rate metric is one of the most important health signals a real-time consumer can emit. The static feed must also be the exact version the real-time feed was generated against, which means version pinning by feed_version and content hash, exactly as the static feed structure guide prescribes. The full join strategy, including frequency-based fallback and unmatched-rate instrumentation, is worked through in matching realtime to static schedules.

import pandas as pd

def match_trip_updates_to_static(
    trip_updates: pd.DataFrame,      # cols: trip_id, stop_id, delay
    stop_times: pd.DataFrame,        # static stop_times.txt
) -> pd.DataFrame:
    stop_times = stop_times.astype({"trip_id": "string", "stop_id": "string"})
    trip_updates = trip_updates.astype({"trip_id": "string", "stop_id": "string"})
    merged = stop_times.merge(
        trip_updates,
        on=["trip_id", "stop_id"],
        how="left",
        validate="one_to_one",
    )
    unmatched = trip_updates.merge(
        stop_times[["trip_id"]].drop_duplicates(), on="trip_id", how="left", indicator=True
    )
    orphan_rate = (unmatched["_merge"] == "left_only").mean()
    if orphan_rate > 0.05:
        logging.warning("high unmatched trip_id rate: %.1f%%", orphan_rate * 100)
    return merged

Common Failure Modes

These are the errors that actually reach production in real-time consumers — the ones that pass a naive smoke test but silently corrupt predictions under real feed conditions:

• Stale feeds served as fresh — the endpoint keeps returning 200 OK with an unchanged header.timestamp because the agency’s producer has frozen. Predictions look valid but are minutes old. Always compare header.timestamp against wall-clock time and treat a feed older than a few polling intervals as degraded, not authoritative.
• Unmatched trip_id — the real-time vehicle.trip.trip_id has no row in the loaded trips.txt because the static feed was republished with regenerated identifiers. The entity is silently dropped and the vehicle vanishes from the map. Pin the static feed version and monitor the unmatched rate.
• Protobuf version drift — decoding a feed produced against a newer gtfs_realtime_version with old gtfs_realtime_pb2 bindings. New fields decode as unknown and are dropped; occasionally the schema is incompatible enough that ParseFromString raises. Keep gtfs-realtime-bindings current and log header.gtfs_realtime_version on every decode.
• Timezone and POSIX-timestamp confusion — treating a TripUpdate’s POSIX arrival.time as if it were a local clock value, or comparing it against a raw stop_times.txt string without first resolving that string through the agency’s IANA timezone. The result is delay figures off by the UTC offset, or by an hour across a DST boundary.
• Entity churn and false disappearance — in a FULL_DATASET (non-incremental) feed, an entity absent from the current message means that trip is simply not currently reporting, not that it was cancelled. Deleting UI state on every absence causes vehicles to flicker. Age out state on a timeout instead of on single-tick absence, and honor incrementality when it is DIFFERENTIAL.

Automation and Python Tooling

The real-time layer leans on a small, focused set of libraries. Unlike static ingestion, where the choice of parser dominates, the real-time stack is a decode step plus standard data tooling:

A production real-time consumer follows this loop on each polling cycle:

1. Issue a conditional GET (with the prior ETag) to the endpoint; on 304 Not Modified, skip the cycle
2. Validate the response content-type and length; decode response.content into a FeedMessage and log gtfs_realtime_version and header.timestamp
3. Reject the message as stale if header.timestamp lags wall-clock by more than a configured threshold
4. Iterate feed_message.entity, routing each by HasField('vehicle'), HasField('trip_update'), or HasField('alert') into three typed collections
5. Flatten each collection into a DataFrame, encoding absent optional fields as NaN rather than zero
6. Join TripUpdates and VehiclePositions to the resident static index on trip_id; forward-fill delay along stop_sequence; expand alert informed_entity scope to affected stops
7. Emit the unmatched-trip_id rate, stale-feed flag, and decode-error count as first-class metrics
8. Publish the merged predictions to the serving layer, and age out any trip or vehicle state not seen within the timeout window

What Robust Real-Time Integration Looks Like

A dependable live layer treats the real-time feed as untrusted, versioned input rather than a source of truth. It pins the exact static feed the stream was generated against, distinguishes absent protobuf fields from zero values, converts every POSIX timestamp through an explicit UTC-to-local step, and instruments unmatched-trip and stale-feed rates as operational signals that gate what riders see. The static feed architecture is the reference baseline; the real-time entities are the moving overlay; and the reconciliation join between them is the component that decides whether your ETAs are trustworthy. From here, the five focused guides below drill into each stage — start with decoding if you are standing up a consumer from scratch, or jump to matching if your entities already decode but your predictions are drifting.

Frequently Asked Questions

Related

• Decoding GTFS-Realtime Protobuf Feeds — polling, conditional requests, and turning a binary FeedMessage into typed entities
• Tracking Vehicle Positions in Realtime — position handling, deriving speed and bearing, mapping vehicles to their stop sequence
• Processing TripUpdates and Delays — delay forward-fill, absolute-versus-relative timing, and computing arrival predictions
• Handling GTFS Service Alerts — active-period filtering, informed-entity scope expansion, and translated text selection
• Matching Realtime to Static Schedules — the trip_id join, frequency-based fallback, and unmatched-rate instrumentation
• GTFS Feed Architecture & Fundamentals — cross-guide: the static baseline the real-time layer is merged against

← Home