Detecting Duplicate Trips in GTFS Feeds

To find duplicate trips, group stop_times.txt by trip_id, build an ordered tuple of (stop_id, arrival_time) for each trip after sorting by stop_sequence, hash that tuple into a single digest, and flag any trips that share a digest within the same service_id. Two trips with different trip_id values but an identical stop-and-time pattern on the same service days are redundant — they inflate per-route frequency counts and mislead any headway or capacity analysis built on top of the feed.

Root Cause Analysis

Duplicate trips almost never come from an agency deliberately scheduling the same bus twice. They come from the pipeline that produced the feed. Understanding the source is what separates a real defect from a benign artefact.

Join fan-out during export is the most common cause. When an agency’s scheduling database builds stop_times.txt by joining a trip pattern to a calendar, a many-to-one relationship that should collapse can instead multiply rows, emitting the same trip pattern under two or more trip_id values. The feed then reports twice the true departures on that corridor, and any frequency calculation reads it as double the real service.

Merged or re-exported feeds introduce duplicates when the same trip survives in two source files that are later combined, or when an incremental export appends trips that already exist. If you assemble regional feeds from several operators, the merge step can reintroduce a pattern that was already present, especially where operators share a corridor.

Block modelling is the benign case that looks identical to the naked eye. A vehicle block_id can legitimately run the same pattern more than once in a day, and interlining can produce trips that share a stop sequence but differ in their surrounding block context. The detector must therefore surface candidates rather than delete anything: the deciding factor is whether the duplicated pattern shares the same service_id and lacks a distinguishing block_id, which is a strong signal of an ETL fault rather than intended repetition.

The reliable discriminator across all three cases is the ordered stop-and-time signature of a trip. If two trips visit the same stops in the same order at the same times, and are active on the same service days, they are — for every practical analytical purpose — the same trip counted twice.

Production-Ready Python Implementation

The script sorts stop_times.txt by (trip_id, stop_sequence), folds each trip into an ordered signature, hashes it with SHA-1, joins the service_id from trips.txt, and reports groups of two or more trips that share both a hash and a service_id. Time columns stay as strings so overnight values above 24:00:00 survive intact.

import hashlib
import logging
from pathlib import Path

import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")


def _trip_fingerprint(group: pd.DataFrame) -> str:
    """Hash a trip's ordered (stop_id, arrival_time) sequence into a digest."""
    ordered = group.sort_values("stop_sequence")
    signature = "|".join(
        f"{stop_id}@{arrival}"
        for stop_id, arrival in zip(ordered["stop_id"], ordered["arrival_time"])
    )
    return hashlib.sha1(signature.encode("utf-8")).hexdigest()


def detect_duplicate_trips(feed_dir: str) -> pd.DataFrame:
    """
    Detect duplicate trips in a GTFS feed.

    A duplicate is a set of trips that share an identical ordered
    (stop_id, arrival_time) sequence AND the same service_id.

    Returns a DataFrame with one row per duplicate group.
    """
    feed_path = Path(feed_dir)

    trips = pd.read_csv(
        feed_path / "trips.txt",
        dtype={"trip_id": str, "route_id": str, "service_id": str, "block_id": str},
        usecols=lambda c: c in {"trip_id", "route_id", "service_id", "block_id"},
    )
    stop_times = pd.read_csv(
        feed_path / "stop_times.txt",
        dtype={
            "trip_id": str,
            "stop_id": str,
            "arrival_time": str,     # keep as str: GTFS allows times past 24:00:00
            "stop_sequence": "Int64",
        },
        usecols=["trip_id", "stop_id", "arrival_time", "stop_sequence"],
    )

    # One digest per trip from its ordered stop-and-time signature.
    fingerprints = (
        stop_times
        .groupby("trip_id", sort=False)
        .apply(_trip_fingerprint, include_groups=False)
        .rename("trip_hash")
        .reset_index()
    )

    # Attach route_id, service_id, block_id for grouping and reporting.
    labelled = fingerprints.merge(trips, on="trip_id", how="left")

    # Duplicates: same hash AND same service_id, appearing more than once.
    grouped = (
        labelled
        .groupby(["trip_hash", "service_id"], sort=False)
        .agg(
            trip_count=("trip_id", "size"),
            trip_ids=("trip_id", lambda s: sorted(s)),
            route_ids=("route_id", lambda s: sorted(set(s))),
            block_ids=("block_id", lambda s: sorted(set(s.dropna()))),
        )
        .reset_index()
    )
    duplicates = grouped[grouped["trip_count"] > 1].copy()

    logging.info(
        "Scanned %d trips | %d duplicate groups covering %d trips",
        len(labelled),
        len(duplicates),
        int(duplicates["trip_count"].sum()) if not duplicates.empty else 0,
    )

    report_path = feed_path / "duplicate_trips.csv"
    duplicates.sort_values("trip_count", ascending=False).to_csv(report_path, index=False)
    logging.info("Duplicate-trip report written to %s", report_path)
    return duplicates


if __name__ == "__main__":
    report = detect_duplicate_trips("/data/gtfs/citybus")
    print(report.to_string(index=False))

Step-by-Step Walkthrough

The fingerprint is order-sensitive. _trip_fingerprint sorts by stop_sequence before joining, so two trips that visit the same stops in a different order produce different digests — as they should, since a reversed route is a genuinely different trip. Encoding both stop_id and arrival_time into the signature means trips that share a path but run at different times are not falsely flagged.

SHA-1 collapses a variable-length sequence to a fixed key. A trip may have three stops or three hundred; hashing turns each into one comparable string. Grouping on that digest finds all collisions in a single linear pass, avoiding the quadratic cost of comparing every trip pattern against every other.

service_id is part of the grouping key. Two trips with the same pattern that run on genuinely different service days — a weekday version and a Saturday version — are not duplicates; they are distinct scheduled service. Requiring an identical service_id before flagging is what keeps the detector from raising false positives on legitimately parallel calendars.

block_id is carried but not used to group. It is reported so a reviewer can immediately see whether a flagged group spans distinct blocks (often legitimate interlining) or shares a block context (more likely an export fault). This turns the CSV into a triage tool rather than a verdict.

Times remain strings throughout. arrival_time is read as str and never parsed, so overnight values such as 25:15:00 are hashed exactly as written. Parsing them to clock times before fingerprinting would corrupt the signature; see the frequency-based versus timetable schedules guide for why frequency-defined trips also need care here.

Verification and Output

Confirm the detector’s output is internally consistent before acting on it.

import pandas as pd
from pathlib import Path

report = pd.read_csv(
    Path("/data/gtfs/citybus") / "duplicate_trips.csv",
    dtype={"trip_hash": str, "service_id": str},
)

# Every reported group must contain at least two trips
assert (report["trip_count"] >= 2).all(), "A group with count < 2 leaked into the report"

# The (trip_hash, service_id) pair must be unique per row
assert not report.duplicated(subset=["trip_hash", "service_id"]).any(), \
    "Duplicate grouping key in report — grouping logic is wrong"

total_redundant = int((report["trip_count"] - 1).sum())
print(f"{len(report)} duplicate groups; {total_redundant} redundant trips could be removed")

The output CSV has one row per duplicate group, sorted so the worst offenders appear first. Each row lists the shared trip_hash, the service_id, how many trips collide, and the specific trip_ids, route_ids, and block_ids involved. A clean feed produces an empty report. The total_redundant figure — the count beyond the first trip in each group — is the number of departures that would vanish from your frequency totals once the duplicates are resolved.

Gotchas and Edge Cases

• Frequency-defined trips have template stop_times.txt. When a trip is driven by frequencies.txt, its stop times are offsets from start_time, not literal departures, so two frequency trips can share an identical template without being redundant service. Detect and exclude frequency-based trips before fingerprinting; the converting frequencies.txt to exact departure times guide explains how the template expansion works.

• Near-duplicates differing by a few seconds. ETL rounding can leave two copies of a trip whose arrival_time values differ by one or two seconds at a single stop. An exact hash treats these as distinct. If you need to catch them, normalise times to a coarser resolution (for example whole minutes) before hashing — at the cost of occasionally merging genuinely close departures.

• Shape-only differences. Two trips may share an identical (stop_id, arrival_time) sequence but reference different shape_id geometries. They are duplicates for scheduling purposes but not for map rendering. Decide per use case whether shape_id belongs in the fingerprint; for headway analysis it should not.

• Empty or single-stop trips. A trip with zero stop_times.txt rows produces an empty signature and can collide with other empty trips. Filter trips to those with at least two stops before fingerprinting so malformed trips are handled by the missing stop_times repair workflow rather than being mislabelled as duplicates.

Frequently Asked Questions

Related

• Merging Multiple GTFS Feeds in Python — a frequent source of the overlapping trips this detector catches
• Calculating Service Headways from GTFS — why duplicate trips corrupt frequency and headway metrics
• Fixing Missing stop_times.txt Records in Python — the complementary case of trips with too few rows rather than too many
• Up: Error Logging and Data Quality Categorization — where duplicate detection sits in a data-quality taxonomy
• Section: Python Parsing & Data Normalization — the full GTFS parsing and normalization pipeline · Home