Agency Metadata and Feed Versioning Practices

When Python pipelines ingest, transform, or distribute GTFS datasets, every route, trip, and stop traces back to two small but load-bearing files: agency.txt and feed_info.txt. Sloppy handling — missing agency_id values, unvalidated timezone strings, no version tags — does not fail loudly at parse time. Instead, the damage surfaces later: routing engines apply the wrong schedule window, fare attribution silently diverges across operators, and audits cannot reconstruct which feed version was live on a given date.

The pages in this section cover the schema constraints, Python validation patterns, and versioning strategies that keep agency metadata trustworthy across the full feed lifecycle in a Python Parsing & Data Normalization context.

Prerequisites

• Python 3.9+ with csv, zipfile, hashlib, datetime, logging from the standard library
• pydantic v2 for strict schema enforcement (pip install pydantic)
• A raw GTFS archive (.zip) containing at minimum agency.txt; feed_info.txt is strongly recommended
• Familiarity with Python Parsing & Data Normalization pipeline patterns

Core Concepts

agency.txt — the ownership anchor

agency.txt defines the legal operating entity behind every route. The GTFS spec marks agency_id as conditionally required: optional when the feed contains exactly one agency, mandatory when two or more are present. In practice, omitting agency_id even in single-agency feeds breaks any downstream merge, so production pipelines should treat it as required unconditionally.

agency_timezone is the most operationally critical field. GTFS departure times are wall-clock local times, not UTC. A wrong or misspelled IANA string (US/Eastern instead of America/New_York) produces silently incorrect departures at DST transitions.

feed_info.txt — provenance and validity window

feed_info.txt records the publisher, language, and the date range during which the feed is valid. Routing engines use feed_start_date and feed_end_date to decide whether a feed is current; data lakes use feed_version to differentiate successive releases.

Pydantic Validation

Define strict models that enforce schema at ingestion time:

import re
from typing import Optional
import pydantic
from pydantic import Field, field_validator
import zoneinfo

VALID_IANA_TIMEZONES = zoneinfo.available_timezones()

class AgencyMetadata(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(str_strip_whitespace=True)

    agency_id: str
    agency_name: str
    agency_url: pydantic.AnyHttpUrl
    agency_timezone: str
    agency_lang: str = Field(pattern=r"^[a-z]{2,3}$")
    agency_phone: Optional[str] = None

    @field_validator("agency_timezone")
    @classmethod
    def validate_iana_timezone(cls, v: str) -> str:
        if v not in VALID_IANA_TIMEZONES:
            raise ValueError(
                f"agency_timezone '{v}' is not a valid IANA timezone. "
                "Common mistake: use 'America/New_York', not 'US/Eastern'."
            )
        return v

class FeedInfoMetadata(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(str_strip_whitespace=True)

    feed_publisher_name: str
    feed_publisher_url: pydantic.AnyHttpUrl
    feed_lang: str = Field(pattern=r"^[a-z]{2,3}$")
    feed_start_date: str = Field(pattern=r"^\d{8}$")
    feed_end_date: str = Field(pattern=r"^\d{8}$")
    feed_version: str

The field_validator on agency_timezone cross-references the zoneinfo standard library to catch the common US/Eastern / America/New_York confusion before it corrupts timezone normalization downstream.

Deterministic SHA-256 Checksums

Always hash the raw .zip before extraction. Middleware layers (antivirus, CDN) sometimes silently normalise line endings in text files after download; hashing the archive detects that drift.

import hashlib
from pathlib import Path

def compute_sha256(file_path: Path) -> str:
    """Stream the raw archive in 8 kB chunks."""
    sha256 = hashlib.sha256()
    with open(file_path, "rb") as fh:
        for chunk in iter(lambda: fh.read(8192), b""):
            sha256.update(chunk)
    return sha256.hexdigest()

For content-addressed hashing that normalises CSV ordering and whitespace before digesting — so two exports of identical schedule data produce the same hash regardless of export metadata — see automating GTFS version control with Python scripts.

Versioning Strategies

Two schemes work well in practice:

Date-based tags (YYYYMMDD) align with feed_start_date and sort lexicographically. They communicate when a feed was published but give no indication of how much changed.

Content-hash tags (sha256:abc123) are cryptographically immutable. Two feeds with identical normalized bytes share one hash; any substantive change produces a new hash. This integrates cleanly with CI pipelines that gate routing-engine rebuilds on hash changes.

For teams that need both properties, combine them: metro-central/20260624/3f8a1c4e7d2b. The agency prefix scopes the tag namespace, the date provides human-readable ordering, and the hash suffix provides content integrity.

Common Failure Modes

• feed_version reused across structural changes. An agency can publish v3 twice in a row with completely different stop coordinates. Never treat feed_version as a stability guarantee — always pair it with a content hash.

• UTF-8 BOM prefix on agency.txt. Some agency tools export with a byte-order mark. Use encoding="utf-8-sig" in pd.read_csv to strip it transparently; otherwise the first column name gains an invisible  prefix that breaks all field lookups.

• Whitespace in timezone strings. " America/New_York" (leading space) passes regex length checks but fails IANA lookup. The str_strip_whitespace=True Pydantic config option removes this class of error without custom validators.

• feed_info.txt absent from the archive. The spec marks it as conditionally required. When missing, fall back to a date-stamped version tag and log a WARNING so downstream consumers know provenance data is incomplete.

Validation and Verification

Metadata is the smallest part of a GTFS feed and the part everything else depends on. Four checks cover what actually goes wrong, and all four run in milliseconds because the tables involved have single-figure row counts.

from zoneinfo import ZoneInfo, ZoneInfoNotFoundError


def verify_metadata(agency, feed_info, routes) -> list[str]:
    problems = []

    # 1. Every agency declares a resolvable IANA timezone. Without it, every
    #    time in the feed is uninterpretable.
    for row in agency.itertuples(index=False):
        try:
            ZoneInfo(str(row.agency_timezone))
        except (ZoneInfoNotFoundError, ValueError):
            problems.append(f"agency {row.agency_id}: "
                            f"{row.agency_timezone!r} is not an IANA zone")

    # 2. agency_id is present whenever more than one agency exists.
    if len(agency) > 1 and agency["agency_id"].isna().any():
        problems.append("a multi-agency feed leaves agency_id blank")

    # 3. Every route attributes to an agency that exists.
    known = set(agency["agency_id"].dropna())
    if known:
        unknown = set(routes["agency_id"].dropna()) - known
        if unknown:
            problems.append(f"routes attribute to unknown agencies: {sorted(unknown)[:3]}")

    # 4. feed_info's declared window is consistent with itself.
    if not feed_info.empty:
        start = feed_info["feed_start_date"].iloc[0]
        end = feed_info["feed_end_date"].iloc[0]
        if start and end and end < start:
            problems.append(f"feed_info window runs backwards: {start}..{end}")

    return problems

The third check earns its place in merged feeds. When two agencies’ data is combined without namespacing, agency_id values collide and routes silently reattribute to the wrong operator — which changes the timezone every one of their trips is interpreted in. The check costs nothing and catches a defect whose symptoms appear hours downstream as schedules that are consistently an hour or several out.

The fourth is a reminder rather than a gate: feed_info.txt’s dates are a statement of intent, not a measurement. The feed’s real coverage is whatever the calendar files contain, which is why service gap and expiry detection reads the calendar rather than this table. Where the two disagree, the calendar wins and the disagreement is worth reporting to the publisher.

Performance and Scale Notes

Nothing here is expensive on one feed. agency.txt has one to a handful of rows, feed_info.txt has at most one, and the Pydantic validation over both completes in microseconds. The cost appears in two other places.

Checksumming. Hashing an archive means reading every byte of it. For a 600 MB feed that is bounded by disk throughput, around two seconds on an SSD, and it must be done over the sorted member names and contents rather than over the archive bytes — two zips of identical content have different bytes, so hashing the file makes every rebuild look like a change. Hashing the sorted members makes the digest a property of the data.

Batch runs. Across forty agencies the metadata checks remain trivial, but forty checksums is forty full archive reads: roughly 80 seconds of pure IO. Two things make that disappear. Compute the digest during the download stream rather than in a second pass, so the bytes are hashed as they arrive and nothing is read twice. And skip the work entirely when the server answers a conditional request with 304 — an unchanged feed needs no digest, because it has already been processed under one. That is the same conditional-fetch discipline the scheduled refresh relies on, and across a large portfolio it turns the versioning stage from minutes into seconds.

Storage is not a consideration. A version record is a digest, a timestamp, a coverage window and a handful of counts — a few hundred bytes per publication, so a decade of weekly history for a hundred agencies fits comfortably in a single table and never needs expiring. Keeping that history indefinitely is what makes it possible to answer the question agencies actually ask, which is not “is this feed valid” but “what changed since last time”.

Frequently Asked Questions

Failure Modes and Edge Cases

• A feed whose agency.txt names an agency that no longer exists. Operators merge and rebrand, and the identifier usually outlives the name. Keep the historical agency_id rather than renaming it, because every archived version of the feed references it.
• feed_info.txt absent entirely. Legal, and common. Everything the file would have told you — the publisher, the version, the coverage window — has to come from elsewhere: the fetch URL, the content digest, and the calendar files respectively.
• Two publications with identical content and different declared versions. The digest is the same, so no new version should be recorded. Trusting feed_version here creates a history full of entries that differ in nothing but a string.
• A feed republished with the same declared version and different content. The opposite case, and much more dangerous: any cache keyed on feed_version serves stale data indefinitely. This is precisely why the content digest is the version key and the declared version is metadata.
• Multi-agency feeds with mixed languages. agency_lang is per agency, and text handling — casing, sorting, collation of stop names — should follow the agency that owns the route rather than a feed-wide default. Getting this wrong produces stop lists that sort inexplicably in one part of the network.
• Attribution requirements in the licence. agency_url exists partly so a consumer can attribute the data. Dropping the column because nothing in the pipeline reads it is a licence-compliance risk rather than a data-quality one, which is a different conversation and a more expensive one.

In This Section

• Automating GTFS Version Control with Python Scripts — hash-backed snapshots, Git tagging, and idempotent update pipelines

Related

• Automating Feed Updates with GTFS-Kit — HTTP ETag and Last-Modified header detection for new feed publications
• Memory-Efficient Processing for Large Feeds — chunked reading and Parquet output
• GTFS Validation Rules and Common Schema Errors — referential integrity checks before metadata extraction
• Timezone Handling and Schedule Normalization — IANA timezone resolution and DST-safe UTC conversion
• Agency Metadata and Feed Versioning (GTFS Architecture) — the corresponding architecture-layer reference for this topic

Up: Python Parsing & Data Normalization | Home