1. Home
  2. GTFS Realtime Integration
  3. Decoding GTFS Realtime Protobuf Feeds
  4. Parsing GTFS Rt Protobuf With Python

Parsing GTFS-RT Protobuf in Python

Create a gtfs_realtime_pb2.FeedMessage(), call feed.ParseFromString(response.content) on the raw bytes, then iterate feed.entity and read each optional payload only after entity.HasField(...) confirms it is set. Coalesce unset protobuf scalars to None as you extract them, collect the results as plain dict rows, and build one typed pandas DataFrame at the end. The complete script below does exactly that for a vehicle-position feed and is safe to copy into a pipeline.

Root Cause Analysis

The errors people hit when parsing GTFS-Realtime almost never come from protobuf itself — they come from three false assumptions about how the decoded object behaves.

The first is expecting ParseFromString to return the parsed message. It does not. It mutates the receiver in place and returns a byte count. Code written as feed = gtfs_realtime_pb2.FeedMessage().ParseFromString(data) binds feed to an integer, and every subsequent field access raises an AttributeError that looks baffling until you realize the object was thrown away.

The second is treating unset optional fields as if they were null. In protobuf, reading an optional message field that was never set returns a fresh, all-default instance rather than raising. So entity.vehicle always “works,” and a loop that reads it unconditionally happily produces rows full of empty strings and zeros for entities that were actually trip updates or alerts. The HasField guard is the only reliable presence test.

The third is confusing scalar defaults with missing data. An unset string is "", an unset number is 0, an unset enum is its zero member. None of these is None. If you write these straight into a DataFrame you get columns where “absent” and “genuinely zero” are indistinguishable, which quietly corrupts any later filter or join. The fix is to coalesce at extraction time and lean on pandas nullable dtypes. This page implements all three fixes in one script; the broader message model is covered in decoding GTFS-Realtime protobuf feeds.

Production-Ready Python Implementation

The following script is complete and runnable. Point FEED_URL at any GTFS-Realtime vehicle-position endpoint. It fetches, decodes, iterates with guards, and returns a typed frame.

python 
import logging
from typing import Optional

import requests
import pandas as pd
from google.transit import gtfs_realtime_pb2
from google.protobuf.message import DecodeError

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

FEED_URL = "https://example-agency.gov/gtfs-realtime/vehicle-positions.pb"

VEHICLE_DTYPES = {
    "entity_id": "string",
    "vehicle_id": "string",
    "trip_id": "string",
    "route_id": "string",
    "direction_id": "Int64",
    "latitude": "float64",
    "longitude": "float64",
    "bearing": "float64",
    "speed": "float64",
    "current_stop_sequence": "Int64",
    "stop_id": "string",
    "vehicle_timestamp": "Int64",
}


def _str_or_none(value: str) -> Optional[str]:
    """Coalesce an unset protobuf string ('') to a real None."""
    return value or None


def fetch_and_parse(url: str, timeout: float = 15.0) -> gtfs_realtime_pb2.FeedMessage:
    """GET the endpoint and decode the protobuf body into a FeedMessage."""
    response = requests.get(url, timeout=timeout)
    response.raise_for_status()

    feed = gtfs_realtime_pb2.FeedMessage()
    try:
        feed.ParseFromString(response.content)  # in-place mutation, not a return value
    except DecodeError as exc:
        raise ValueError(
            "Response was not valid GTFS-Realtime protobuf — check the URL "
            "returned bytes (response.content) and not an HTML error page"
        ) from exc

    logging.info(
        "Decoded feed: version=%s, entities=%d, built_at=%d",
        feed.header.gtfs_realtime_version,
        len(feed.entity),
        feed.header.timestamp,
    )
    return feed


def vehicle_positions_to_frame(feed: gtfs_realtime_pb2.FeedMessage) -> pd.DataFrame:
    """Flatten every VehiclePosition entity into a typed DataFrame."""
    rows: list[dict] = []

    for entity in feed.entity:
        # Skip trip_update and alert entities in a combined feed.
        if not entity.HasField("vehicle"):
            continue

        vp = entity.vehicle
        has_position = vp.HasField("position")

        rows.append(
            {
                "entity_id": entity.id,
                "vehicle_id": _str_or_none(vp.vehicle.id),
                "trip_id": _str_or_none(vp.trip.trip_id),
                "route_id": _str_or_none(vp.trip.route_id),
                # direction_id is a scalar with a 0 default — guard it explicitly
                "direction_id": vp.trip.direction_id if vp.trip.HasField("direction_id") else pd.NA,
                "latitude": vp.position.latitude if has_position else pd.NA,
                "longitude": vp.position.longitude if has_position else pd.NA,
                "bearing": vp.position.bearing if has_position and vp.position.HasField("bearing") else pd.NA,
                "speed": vp.position.speed if has_position and vp.position.HasField("speed") else pd.NA,
                "current_stop_sequence": (
                    vp.current_stop_sequence
                    if vp.HasField("current_stop_sequence")
                    else pd.NA
                ),
                "stop_id": _str_or_none(vp.stop_id),
                "vehicle_timestamp": vp.timestamp if vp.timestamp else pd.NA,
            }
        )

    if not rows:
        logging.warning("No vehicle entities found — feed may carry only trip updates or alerts")
        return pd.DataFrame(columns=list(VEHICLE_DTYPES)).astype(VEHICLE_DTYPES)

    frame = pd.DataFrame(rows).astype(VEHICLE_DTYPES)
    frame["observed_at"] = pd.to_datetime(frame["vehicle_timestamp"], unit="s", utc=True)
    logging.info("Built vehicle frame: %d rows, %d with a position",
                 len(frame), int(frame["latitude"].notna().sum()))
    return frame


if __name__ == "__main__":
    feed_message = fetch_and_parse(FEED_URL)
    vehicle_positions = vehicle_positions_to_frame(feed_message)
    print(vehicle_positions.head(10).to_string(index=False))

Step-by-Step Walkthrough

Construct then parse. feed = gtfs_realtime_pb2.FeedMessage() creates an empty message; feed.ParseFromString(response.content) fills it. The two-line separation is deliberate — chaining them onto one line captures the return value (a byte count) instead of the message, which is the single most common beginner mistake.

Read response.content, not response.text. The body is binary. requests would apply a character decode to .text and corrupt the payload, producing a DecodeError that masquerades as a bad feed. The try/except around ParseFromString turns that into an actionable message that also catches an HTML error page served with a 200 status.

Guard the payload with HasField. entity.HasField("vehicle") is the presence test. Because a combined feed interleaves vehicles, trip updates, and alerts, the continue skips non-vehicle entities cleanly rather than emitting all-zero rows for them.

Coalesce scalars at extraction. _str_or_none turns an unset string ("") into None. For scalar numbers whose 0 default is ambiguous — direction_id, current_stop_sequence, bearing, speed — the code checks HasField and substitutes pd.NA, so a genuine zero is never confused with a missing value. The position submessage itself is guarded once with has_position before any of its members are read.

Build one typed frame. Rows accumulate as plain dicts, then a single pd.DataFrame(rows).astype(VEHICLE_DTYPES) materializes the frame. Declaring nullable string and Int64 dtypes keeps route_id like "0012" from becoming the integer 12 and lets integer columns carry pd.NA. The empty-feed branch still returns a correctly typed, zero-row frame so callers never special-case None.

Derive a UTC datetime. pd.to_datetime(..., unit="s", utc=True) converts the per-vehicle POSIX timestamp into a timezone-aware column. Keeping it in UTC at rest matches the timezone handling and schedule normalization rule of localizing only for display.

Verification and Output

Assert the frame’s structure before handing it downstream. These checks catch the exact corruptions the guards are designed to prevent.

python 
def assert_vehicle_frame(vehicle_positions: pd.DataFrame) -> None:
    # entity_id is unique within a single FeedMessage
    assert vehicle_positions["entity_id"].is_unique, "duplicate entity_id in one message"

    # Coordinates, where present, are within WGS84 bounds
    located = vehicle_positions.dropna(subset=["latitude", "longitude"])
    assert located["latitude"].between(-90, 90).all(), "latitude out of bounds"
    assert located["longitude"].between(-180, 180).all(), "longitude out of bounds"

    # Nullable dtypes survived construction
    assert vehicle_positions["route_id"].dtype == "string"
    assert vehicle_positions["current_stop_sequence"].dtype == "Int64"
    print(f"OK: {len(vehicle_positions)} rows, {len(located)} positioned")


assert_vehicle_frame(vehicle_positions)

A representative first row of the printed output looks like this — note the <NA> values where optional fields were genuinely unset rather than zero:

text 
entity_id vehicle_id  trip_id route_id  direction_id  latitude  longitude  bearing  speed  current_stop_sequence stop_id  vehicle_timestamp                observed_at
  veh_1041       1041  4021_12       12             0  52.37031    4.89921    143.0    8.4                      7   S_0093         1715007312 2024-05-06 18:55:12+00:00

Gotchas and Edge Cases

  • direction_id legitimately takes the value 0. GTFS uses 0 and 1 for the two travel directions, so 0 is real data, not an absent field. This is why the script uses HasField("direction_id") rather than a truthiness test — a naive vp.trip.direction_id or None would erase every outbound trip.

  • Combined feeds interleave entity kinds. If your endpoint serves vehicles, trip updates, and alerts together, the HasField("vehicle") filter is essential. Without it you would materialize empty rows for every non-vehicle entity and inflate your row count with junk.

  • vehicle.timestamp may trail header.timestamp. The per-vehicle timestamp reflects when that vehicle last reported, which can be older than when the producer built the snapshot. Keep the vehicle timestamp for freshness reasoning about a specific vehicle and read the header separately when you need the snapshot time.

  • An empty entity list is valid. Overnight or during an outage the feed can return a well-formed message with zero entities. The script returns a typed empty frame instead of raising, so a scheduled job can log the gap and move on rather than crashing.

Frequently Asked Questions

What does ParseFromString return?

Nothing useful — it returns the number of bytes consumed but mutates the FeedMessage object in place. Create the message first with feed = gtfs_realtime_pb2.FeedMessage(), then call feed.ParseFromString(response.content) and read from feed afterward.

Why does entity.vehicle return data even when there is no vehicle?

Protobuf returns a default-constructed empty message for an unset optional field rather than raising or returning null. Guard the access with entity.HasField('vehicle') so you never treat an all-zero default as a real vehicle position.

How do I turn an unset protobuf string into a real null?

An unset protobuf string field reads as an empty string, so the idiom value or None converts it to a Python None. Load the resulting column as the pandas nullable string dtype so empty and missing are represented consistently.