Agency Metadata and Feed Versioning Practices

When Python pipelines ingest, transform, or distribute GTFS datasets, every route, trip, and stop traces back to two small but load-bearing files: agency.txt and feed_info.txt. Sloppy handling — missing agency_id values, unvalidated timezone strings, no version tags — does not fail loudly at parse time. Instead, the damage surfaces later: routing engines apply the wrong schedule window, fare attribution silently diverges across operators, and audits cannot reconstruct which feed version was live on a given date.

The pages in this section cover the schema constraints, Python validation patterns, and versioning strategies that keep agency metadata trustworthy across the full feed lifecycle in a Python Parsing & Data Normalization context.

Prerequisites

• Python 3.9+ with csv, zipfile, hashlib, datetime, logging from the standard library
• pydantic v2 for strict schema enforcement (pip install pydantic)
• A raw GTFS archive (.zip) containing at minimum agency.txt; feed_info.txt is strongly recommended
• Familiarity with Python Parsing & Data Normalization pipeline patterns

Core Concepts

agency.txt — the ownership anchor

agency.txt defines the legal operating entity behind every route. The GTFS spec marks agency_id as conditionally required: optional when the feed contains exactly one agency, mandatory when two or more are present. In practice, omitting agency_id even in single-agency feeds breaks any downstream merge, so production pipelines should treat it as required unconditionally.

agency_timezone is the most operationally critical field. GTFS departure times are wall-clock local times, not UTC. A wrong or misspelled IANA string (US/Eastern instead of America/New_York) produces silently incorrect departures at DST transitions.

feed_info.txt — provenance and validity window

feed_info.txt records the publisher, language, and the date range during which the feed is valid. Routing engines use feed_start_date and feed_end_date to decide whether a feed is current; data lakes use feed_version to differentiate successive releases.

Pydantic Validation

Define strict models that enforce schema at ingestion time:

import re
from typing import Optional
import pydantic
from pydantic import Field, field_validator
import zoneinfo

VALID_IANA_TIMEZONES = zoneinfo.available_timezones()

class AgencyMetadata(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(str_strip_whitespace=True)

    agency_id: str
    agency_name: str
    agency_url: pydantic.AnyHttpUrl
    agency_timezone: str
    agency_lang: str = Field(pattern=r"^[a-z]{2,3}$")
    agency_phone: Optional[str] = None

    @field_validator("agency_timezone")
    @classmethod
    def validate_iana_timezone(cls, v: str) -> str:
        if v not in VALID_IANA_TIMEZONES:
            raise ValueError(
                f"agency_timezone '{v}' is not a valid IANA timezone. "
                "Common mistake: use 'America/New_York', not 'US/Eastern'."
            )
        return v

class FeedInfoMetadata(pydantic.BaseModel):
    model_config = pydantic.ConfigDict(str_strip_whitespace=True)

    feed_publisher_name: str
    feed_publisher_url: pydantic.AnyHttpUrl
    feed_lang: str = Field(pattern=r"^[a-z]{2,3}$")
    feed_start_date: str = Field(pattern=r"^\d{8}$")
    feed_end_date: str = Field(pattern=r"^\d{8}$")
    feed_version: str

The field_validator on agency_timezone cross-references the zoneinfo standard library to catch the common US/Eastern / America/New_York confusion before it corrupts timezone normalization downstream.

Deterministic SHA-256 Checksums

Always hash the raw .zip before extraction. Middleware layers (antivirus, CDN) sometimes silently normalise line endings in text files after download; hashing the archive detects that drift.

import hashlib
from pathlib import Path

def compute_sha256(file_path: Path) -> str:
    """Stream the raw archive in 8 kB chunks."""
    sha256 = hashlib.sha256()
    with open(file_path, "rb") as fh:
        for chunk in iter(lambda: fh.read(8192), b""):
            sha256.update(chunk)
    return sha256.hexdigest()

For content-addressed hashing that normalises CSV ordering and whitespace before digesting — so two exports of identical schedule data produce the same hash regardless of export metadata — see automating GTFS version control with Python scripts.

Versioning Strategies

Two schemes work well in practice:

Date-based tags (YYYYMMDD) align with feed_start_date and sort lexicographically. They communicate when a feed was published but give no indication of how much changed.

Content-hash tags (sha256:abc123) are cryptographically immutable. Two feeds with identical normalized bytes share one hash; any substantive change produces a new hash. This integrates cleanly with CI pipelines that gate routing-engine rebuilds on hash changes.

For teams that need both properties, combine them: metro-central/20260624/3f8a1c4e7d2b. The agency prefix scopes the tag namespace, the date provides human-readable ordering, and the hash suffix provides content integrity.

Common Failure Modes

• feed_version reused across structural changes. An agency can publish v3 twice in a row with completely different stop coordinates. Never treat feed_version as a stability guarantee — always pair it with a content hash.

• UTF-8 BOM prefix on agency.txt. Some agency tools export with a byte-order mark. Use encoding="utf-8-sig" in pd.read_csv to strip it transparently; otherwise the first column name gains an invisible  prefix that breaks all field lookups.

• Whitespace in timezone strings. " America/New_York" (leading space) passes regex length checks but fails IANA lookup. The str_strip_whitespace=True Pydantic config option removes this class of error without custom validators.

• feed_info.txt absent from the archive. The spec marks it as conditionally required. When missing, fall back to a date-stamped version tag and log a WARNING so downstream consumers know provenance data is incomplete.

In This Section

• Automating GTFS Version Control with Python Scripts — hash-backed snapshots, Git tagging, and idempotent update pipelines

Related

• Automating Feed Updates with GTFS-Kit — HTTP ETag and Last-Modified header detection for new feed publications
• Memory-Efficient Processing for Large Feeds — chunked reading and Parquet output
• GTFS Validation Rules and Common Schema Errors — referential integrity checks before metadata extraction
• Timezone Handling and Schedule Normalization — IANA timezone resolution and DST-safe UTC conversion
• Agency Metadata and Feed Versioning (GTFS Architecture) — the corresponding architecture-layer reference for this topic

Up: Python Parsing & Data Normalization | Home