Reconstructing Vehicle Blocks from block_id

Group trips by (block_id, service_id) — never by block_id alone — order each group by first departure time, and then check two things: that no two consecutive trips overlap in time, and that the vehicle could plausibly get from where one trip ended to where the next begins. Overlapping trips in a block are a physical impossibility that no schema validator reports, and they are common in feeds where block assignment comes from a different system than the timetable. The wider linking model is covered in transfers and trip linking.

Root Cause Analysis

block_id is one optional column with one sentence of specification behind it: trips sharing a value are operated by the same vehicle. Everything a pipeline needs beyond that has to be inferred, and each inference is a place to go wrong.

The order is not stated. It comes from the times, which means it comes from stop_times.txt rather than from trips.txt, which means reconstructing blocks requires an aggregation over the feed’s largest table. Implementations that try to avoid that aggregation end up ordering by trip_id, which is alphabetical and meaningless.

The scope is not stated. A block_id is not unique across service days. The same vehicle assignment repeats every weekday, so a feed covering six months contains the same block_id on 130 different dates. Grouping without service_id produces one enormous chain per block, every trip in it overlapping every other, and a validation report consisting entirely of false positives.

Continuity is not guaranteed. Where one trip ends need not be where the next begins. The vehicle deadheads — drives out of service — between them, which is ordinary practice and invisible in the feed. Treating every discontinuity as an error produces noise; treating none of them as an error misses the case where the gap is too short for the distance.

The consequence of getting blocks wrong in the other direction — ignoring them — is subtler and more expensive. A rider on a through-routed service stays in their seat while the vehicle changes trip identity at a terminus. A pipeline without block awareness sees two trips, infers a transfer, adds an interchange penalty and, on a v1 fare model, charges a second fare. On a network that through-routes heavily, that is a large share of all journeys priced and timed wrongly.

Production-Ready Python Implementation

"""Reconstruct GTFS vehicle blocks and check that each one is physically possible."""
from __future__ import annotations

import logging
import math
from dataclasses import dataclass
from pathlib import Path
from zipfile import ZipFile

import pandas as pd

log = logging.getLogger("gtfs.blocks")

# The fastest a service vehicle plausibly deadheads, in metres per second.
# 25 m/s is 90 km/h — generous for a bus, tight for a regional train.
MAX_DEADHEAD_SPEED = 25.0
EARTH_RADIUS_M = 6_371_000.0


@dataclass(frozen=True)
class BlockTrip:
    trip_id: str
    route_id: str
    first_stop: str
    last_stop: str
    start_s: int          # seconds since the start of the service day
    end_s: int


@dataclass
class Block:
    block_id: str
    service_id: str
    trips: list[BlockTrip]

    @property
    def span_s(self) -> int:
        return self.trips[-1].end_s - self.trips[0].start_s


def _seconds(series: pd.Series) -> pd.Series:
    """GTFS H:MM:SS to seconds since the service day started — hours may exceed 24."""
    parts = series.str.strip().str.split(":", expand=True).astype("float64")
    return (parts[0] * 3600 + parts[1] * 60 + parts[2]).astype("Int64")


def read_block_trips(feed_path: Path) -> pd.DataFrame:
    with ZipFile(feed_path) as archive:
        with archive.open("trips.txt") as fh:
            trips = pd.read_csv(fh, dtype="string")
        with archive.open("stop_times.txt") as fh:
            stop_times = pd.read_csv(
                fh, dtype="string",
                usecols=["trip_id", "stop_id", "stop_sequence",
                         "arrival_time", "departure_time"])

    if "block_id" not in trips.columns:
        log.info("%s does not use block_id", feed_path.name)
        return pd.DataFrame()

    stop_times["stop_sequence"] = stop_times["stop_sequence"].astype("int32")
    stop_times["start_s"] = _seconds(stop_times["departure_time"])
    stop_times["end_s"] = _seconds(stop_times["arrival_time"])
    stop_times = stop_times.sort_values(["trip_id", "stop_sequence"])

    bounds = stop_times.groupby("trip_id").agg(
        first_stop=("stop_id", "first"), last_stop=("stop_id", "last"),
        start_s=("start_s", "first"), end_s=("end_s", "last"))

    blocked = trips[trips["block_id"].notna() & (trips["block_id"] != "")]
    return blocked.join(bounds, on="trip_id")


def build_blocks(block_trips: pd.DataFrame) -> list[Block]:
    blocks: list[Block] = []
    if block_trips.empty:
        return blocks
    # A block is one vehicle's work on ONE day, so service_id is part of the key.
    for (block_id, service_id), group in block_trips.groupby(["block_id", "service_id"]):
        ordered = group.sort_values(["start_s", "trip_id"])
        blocks.append(Block(
            block_id=str(block_id), service_id=str(service_id),
            trips=[BlockTrip(trip_id=str(r.trip_id), route_id=str(r.route_id),
                             first_stop=str(r.first_stop), last_stop=str(r.last_stop),
                             start_s=int(r.start_s), end_s=int(r.end_s))
                   for r in ordered.itertuples(index=False)]))
    log.info("reconstructed %d block(s) covering %d trip(s)",
             len(blocks), sum(len(b.trips) for b in blocks))
    return blocks


def _haversine_m(a: tuple[float, float], b: tuple[float, float]) -> float:
    lat1, lon1 = math.radians(a[0]), math.radians(a[1])
    lat2, lon2 = math.radians(b[0]), math.radians(b[1])
    h = (math.sin((lat2 - lat1) / 2) ** 2
         + math.cos(lat1) * math.cos(lat2) * math.sin((lon2 - lon1) / 2) ** 2)
    return 2 * EARTH_RADIUS_M * math.asin(math.sqrt(h))


def check_blocks(blocks: list[Block],
                 stop_coords: dict[str, tuple[float, float]]) -> list[str]:
    problems: list[str] = []
    for block in blocks:
        for earlier, later in zip(block.trips, block.trips[1:]):
            if later.start_s < earlier.end_s:
                overlap = earlier.end_s - later.start_s
                problems.append(
                    f"block {block.block_id}/{block.service_id}: {earlier.trip_id} and "
                    f"{later.trip_id} overlap by {overlap}s — one vehicle cannot run both")
                continue
            if later.first_stop == earlier.last_stop:
                continue                      # continuous, nothing to check
            here = stop_coords.get(earlier.last_stop)
            there = stop_coords.get(later.first_stop)
            if here is None or there is None:
                continue
            metres = _haversine_m(here, there)
            seconds = later.start_s - earlier.end_s
            if seconds <= 0 or metres / seconds > MAX_DEADHEAD_SPEED:
                problems.append(
                    f"block {block.block_id}/{block.service_id}: vehicle must cover "
                    f"{metres / 1000:.1f} km from {earlier.last_stop} to "
                    f"{later.first_stop} in {seconds}s — impossible")
    return problems

Step-by-Step Walkthrough

_seconds parses the clock strings arithmetically, not through a time type. GTFS hours run past 24, so 25:15:00 is a real value that datetime.strptime refuses. Splitting on the colon and multiplying out gives seconds since the service day began, which is the only representation in which two trips in a block compare correctly. The same reasoning drives schedule normalisation throughout the feed.

Bounds come from one aggregation over stop_times.txt. Sorting by (trip_id, stop_sequence) and taking first and last per group gives the four values a block needs — where the trip starts, where it ends, when it leaves and when it arrives — in a single pass over the largest table in the feed. Doing it per block instead would read that table once per block.

The group key is ["block_id", "service_id"]. This is the line that makes the whole function correct. Everything else in the module is arithmetic; this is the modelling decision.

Ordering breaks ties on trip_id. Two trips in a block starting at the same second have no defined order, which is itself a defect — but the sort still needs to be deterministic, or the same feed produces different reports on different runs.

Overlaps and teleports are checked in the same pass but reported differently. An overlap is unambiguous: the times say one vehicle is in two places. A gap between stops is checked against a speed limit, because the vehicle is allowed to deadhead and the only question is whether it could have covered the distance. MAX_DEADHEAD_SPEED is generous on purpose — the goal is to catch a block that jumps across a city in ninety seconds, not to audit running times.

continue after reporting an overlap. A pair that overlaps in time is already broken, and measuring its deadhead speed would produce a second, derived complaint about the same defect.

Verification and Output

def verify(blocks: list[Block], block_trips: pd.DataFrame) -> None:
    seen = sum(len(b.trips) for b in blocks)
    assert seen == len(block_trips), (
        f"reconstructed {seen} trips from {len(block_trips)} blocked trips")

    for block in blocks:
        starts = [t.start_s for t in block.trips]
        assert starts == sorted(starts), f"block {block.block_id} is out of order"
        assert len({t.trip_id for t in block.trips}) == len(block.trips), (
            f"block {block.block_id} contains a duplicate trip")
        assert block.span_s >= 0, f"block {block.block_id} ends before it starts"


def summarise(blocks: list[Block]) -> None:
    lengths = sorted(len(b.trips) for b in blocks)
    spans = sorted(b.span_s / 3600 for b in blocks)
    log.info("blocks: median %d trip(s), longest %d; median span %.1f h, longest %.1f h",
             lengths[len(lengths) // 2], lengths[-1],
             spans[len(spans) // 2], spans[-1])

On a well-formed urban bus feed:

INFO gtfs.blocks: reconstructed 1846 block(s) covering 24310 trip(s)
INFO gtfs.blocks: blocks: median 12 trip(s), longest 41; median span 8.4 h, longest 18.2 h

Those numbers are the sanity check. A median block of twelve trips over eight hours is a plausible driver shift; a median span of 140 hours means the grouping key is missing service_id, and a median of one trip per block means the agency populates block_id uniquely per trip, which conveys nothing.

A feed with the common defect:

ERROR block 4471/WEEKDAY: 1128_1420 and 1131_1425 overlap by 300s — one vehicle cannot run both
ERROR block 4471/WEEKDAY: vehicle must cover 14.2 km from stop_9021 to stop_1187 in 240s — impossible

Gotchas and Edge Cases

block_id present but empty on most trips. Normal. Only the trips that carry a value participate in a block; the rest are standalone. Filtering on both notna() and != "" matters, because an empty string is not null after reading with dtype="string".
Blocks that span midnight. Handled naturally by the seconds-since-service-day representation: a trip ending at 25:40:00 is 92,400 seconds, which is correctly after a trip ending at 23:50:00. Converting to wall-clock time first would break exactly this case.
block_id reused across agencies in a merged feed. Namespacing must happen during the merge, alongside every other identifier. An unprefixed block chains vehicles from different cities and produces spectacular teleport reports.
Stops missing coordinates. The teleport check skips the pair rather than assuming zero distance, which would report every deadhead as instantaneous and therefore fine.
Interlining across modes. A block containing both a bus trip and a rail trip is almost always a defect, and it is worth an explicit check on feeds that carry several route_type values. The vehicle cannot change mode.
Very long blocks on rail feeds. A single unit working eighteen hours is real. Do not cap block length; report the distribution and let a human decide what looks wrong for that operator.

Frequently Asked Questions

Why must blocks be grouped by service_id as well as block_id?

Because a block is one vehicle’s work on one day. Grouping by block_id alone chains together every Monday’s assignment with every Tuesday’s, producing chains of hundreds of trips that no vehicle ran and an overlap report that fires on all of them.

Is a gap between where one trip ends and the next begins an error?

Not by itself. It means the vehicle deadheads — runs out of service between the two points — which is normal operational practice. It becomes an error only when the gap is too short for the vehicle to have covered the distance.

Does GTFS give the order of trips within a block?

No. The specification says only that trips sharing a block_id are run by the same vehicle. The order has to be derived by sorting on each trip’s first departure time, which is why a block containing two trips that start at the same moment has no defined order and is a defect.

What breaks if I ignore blocks entirely?

Riders staying aboard through a trip change get charged and timed as though they had transferred. On networks that through-route heavily — most rail operators and many bus trunk corridors — that is a large fraction of journeys.

Parsing GTFS transfers.txt in Python — the rider-side mechanism, and why it is not the same thing
Detecting Impossible Transfers in GTFS — the same physical-plausibility reasoning applied to riders
Up: Transfers and Trip Linking — both linking mechanisms in context
Section: Python Parsing & Data Normalization · Home