Best Practices for GTFS Agency Metadata

Treat agency.txt as the immutable anchor of your GTFS feed. Assign each agency a stable, human-readable agency_id string, confirm agency_timezone against the IANA Time Zone Database, restrict agency_lang to a two-letter ISO 639-1 code, and enforce https:// on every URL field. Run these checks as a pre-commit hook so broken metadata never reaches downstream consumers.

Root Cause: Why Agency Metadata Breaks Production Pipelines

The GTFS specification marks agency_id, agency_phone, agency_fare_url, agency_email, and agency_lang as optional. In practice, most routing engines, fare calculators, and realtime bridges treat them as required. Omitting or malforming any of these fields causes silent failures:

• Integer agency_id fields are coerced to float64 by pandas when null values appear in adjacent rows, turning "10" into "10.0" and snapping every route–trip–agency foreign-key join.
• Timezone abbreviations such as EST or IST are ambiguous across regions. A routing engine that interprets IST as India Standard Time (UTC+5:30) instead of Irish Standard Time (UTC+1) will produce departure times off by 4.5 hours through every DST boundary — a failure that timezone normalization cannot recover if the raw string is already wrong.
• HTTP URLs in agency_url trigger mixed-content blocks in mobile apps and fail transit API security audits.
• Missing agency_lang causes accessibility tools and screen-reader integrations to fall back to the system locale, which breaks multilingual transit interfaces.

These are not edge cases — they appear in feeds from regional operators, university shuttle systems, and legacy municipal exports that were authored before the spec tightened its validation guidance.

The agency.txt Data Model and Common Failure Modes

agency.txt is a flat CSV with one row per operating entity. Every route in routes.txt carries an agency_id foreign key that must resolve to exactly one row in agency.txt. In single-agency feeds the field is technically optional, but any automated pipeline that later merges feeds will silently corrupt joins if the column was omitted.

Production-Ready Python Implementation

The following module validates agency.txt using Pydantic v2 for schema enforcement and pandas for CSV ingestion. It normalizes inputs, rejects malformed records with field-level error messages, and returns a clean list of validated records suitable for database insertion or feed merging.

import pandas as pd
from pydantic import BaseModel, Field, field_validator, ValidationError
from typing import Optional
import zoneinfo
import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")
logger = logging.getLogger("gtfs.agency_validator")

REQUIRED_COLUMNS = {
    "agency_id", "agency_name", "agency_url", "agency_timezone", "agency_lang"
}

class AgencyRecord(BaseModel):
    agency_id: str = Field(..., min_length=2, max_length=50)
    agency_name: str = Field(..., min_length=2)
    agency_url: str = Field(..., pattern=r"^https://")
    agency_timezone: str
    agency_lang: str = Field(..., min_length=2, max_length=2)
    agency_phone: Optional[str] = None
    agency_fare_url: Optional[str] = None
    agency_email: Optional[str] = None

    @field_validator("agency_id")
    @classmethod
    def reject_numeric_id(cls, v: str) -> str:
        if v.lstrip("-").isdigit():
            raise ValueError(
                f"agency_id '{v}' is numeric — use a stable string like 'MTA-NYCT'. "
                "Integers coerce to float64 in pandas and break downstream joins."
            )
        return v

    @field_validator("agency_timezone")
    @classmethod
    def validate_timezone(cls, v: str) -> str:
        try:
            zoneinfo.ZoneInfo(v)
            return v
        except zoneinfo.ZoneInfoNotFoundError:
            raise ValueError(
                f"'{v}' is not a valid IANA timezone. "
                "Use identifiers like 'America/New_York', not abbreviations like 'EST'."
            )

    @field_validator("agency_lang")
    @classmethod
    def validate_language(cls, v: str) -> str:
        if not v.isalpha() or not v.islower() or len(v) != 2:
            raise ValueError(
                f"agency_lang '{v}' is not a valid ISO 639-1 code. "
                "Use two lowercase letters, e.g. 'en', 'es', 'fr'."
            )
        return v

    @field_validator("agency_fare_url")
    @classmethod
    def fare_url_https(cls, v: Optional[str]) -> Optional[str]:
        if v and not v.startswith("https://"):
            raise ValueError(f"agency_fare_url must use https://: got '{v}'")
        return v


def validate_agency_csv(filepath: str) -> list[dict]:
    """
    Read agency.txt, enforce schema, and return a list of clean record dicts.

    Raises ValueError if no valid rows survive validation (feed is rejected).
    """
    df = pd.read_csv(
        filepath,
        dtype=str,            # prevent integer/float coercion of agency_id
        keep_default_na=False # treat empty cells as '' not NaN
    )

    missing = REQUIRED_COLUMNS - set(df.columns)
    if missing:
        raise ValueError(f"agency.txt is missing required columns: {missing}")

    # Strip leading/trailing whitespace introduced by some agency export tools
    df = df.apply(lambda col: col.str.strip() if col.dtype == object else col)

    valid_records: list[dict] = []
    for idx, row in df.iterrows():
        try:
            record = AgencyRecord(**row.to_dict())
            valid_records.append(record.model_dump())
        except ValidationError as exc:
            for err in exc.errors():
                field = ".".join(str(loc) for loc in err["loc"])
                logger.warning("Row %d | %s | %s", idx, field, err["msg"])

    if not valid_records:
        raise ValueError(
            "agency.txt: no valid records survived validation. Feed rejected."
        )

    logger.info("agency.txt: %d/%d records passed validation.", len(valid_records), len(df))
    return valid_records

Step-by-Step Walkthrough

dtype=str on read (line 46). Without this, pandas infers column types. An agency_id column that contains only digits — common in legacy municipal exports — becomes float64, turning "10" into 10.0. Every subsequent join against routes.txt will fail silently if routes.agency_id was read as str.

keep_default_na=False (line 47). pandas maps blank strings and tokens like "NA", "N/A", and "NULL" to NaN by default. For a column like agency_phone, that may be intentional — but for agency_id a value of NaN will pass the column-presence check and blow up downstream. Disabling this forces explicit handling of every blank.

df.apply(lambda col: col.str.strip() ...) (line 52). Many agency export tools pad fields with whitespace. A timezone value of " America/New_York" (leading space) fails zoneinfo.ZoneInfo() lookup and produces a cryptic error. Stripping the whole frame before validation is the cheapest fix.

reject_numeric_id validator. The spec says agency_id is a unique identifier; it does not restrict its type. This validator enforces the string-only rule explicitly rather than relying on the Pydantic str type annotation, which would silently accept "10".

zoneinfo.ZoneInfo(v) (line 69). The zoneinfo module is part of the Python standard library since 3.9. It resolves against the system’s IANA timezone database (and falls back to the tzdata backport package on platforms without a system database). If lookup fails, the exact invalid string is surfaced in the error message, making log triage fast.

Field-level error extraction (line 86). Pydantic ValidationError.errors() returns a list of dicts with loc (a tuple of field names) and msg. Flattening the location tuple into a dot-separated string makes log lines directly grep-able: Row 4 | agency_timezone | 'EST' is not a valid IANA timezone.

Verification and Output

After running validate_agency_csv("agency.txt"), confirm correctness with:

records = validate_agency_csv("agency.txt")

# Confirm all agency_id values are unique — duplicates break route joins
agency_ids = [r["agency_id"] for r in records]
assert len(agency_ids) == len(set(agency_ids)), (
    f"Duplicate agency_id values: {[i for i in agency_ids if agency_ids.count(i) > 1]}"
)

# Confirm every timezone resolves in the current environment
import zoneinfo
for r in records:
    tz = zoneinfo.ZoneInfo(r["agency_timezone"])
    assert tz is not None, f"Timezone lookup returned None for {r['agency_timezone']}"

# Spot-check: print the first record's key fields
print({k: records[0][k] for k in ("agency_id", "agency_timezone", "agency_lang")})
# Expected: {'agency_id': 'MTA-NYCT', 'agency_timezone': 'America/New_York', 'agency_lang': 'en'}

Expected log output for a clean two-agency feed:

INFO gtfs.agency_validator: agency.txt: 2/2 records passed validation.

A feed with one malformed row:

WARNING gtfs.agency_validator: Row 1 | agency_timezone | 'EST' is not a valid IANA timezone. Use identifiers like 'America/New_York', not abbreviations like 'EST'.
INFO gtfs.agency_validator: agency.txt: 1/2 records passed validation.

Gotchas and Edge Cases

Feeds with a single agency and no agency_id column. The GTFS spec permits omitting agency_id entirely when there is only one operating entity. The code above will raise ValueError because agency_id is in REQUIRED_COLUMNS. For single-agency feeds, add a pre-check: if the column is absent and len(df) == 1, synthesize a slug from agency_name and continue. Never allow a missing column to silently pass without a log warning.

Multi-agency feeds and agency_id collisions. When merging regional feeds — for example, combining BART and Muni into a Bay Area consolidated package — agency_id values from each source feed will collide unless namespaced before concatenation. Apply a prefix such as SFMTA_MUNI and SFMTA_BART to every agency row before concatenation, then re-validate. Maintain a crosswalk table that maps legacy identifiers to canonical namespaced ones; without it, automating feed version diffs becomes impossible because the same operator appears under two different IDs depending on feed origin.

agency_lang with BCP-47 extended tags. Some agencies export values like en-US or zh-Hant. The two-letter-only validator above will reject these. Whether to accept them depends on your consumer stack — GTFS-consuming libraries often strip the subtag anyway. Document the decision: if you accept extended tags, update the validator’s regex and add a note in your feed changelog.

Timezone drift during DST transitions. Even a correctly specified IANA timezone can produce wrong departure times if the feed export tool applies the UTC offset at export time rather than at runtime. This is distinct from a malformed agency_timezone field — it is a schedule interpolation error. See converting local transit times to UTC in Python for the correct approach to resolving wall-clock departure times against IANA zone rules.

agency_url redirects. A URL that resolves after one or two 301/302 redirects will pass the https:// prefix check but may silently redirect to an HTTP endpoint at the final hop. For production validation, add a lightweight requests.head() check that follows redirects and asserts the final URL is HTTPS and returns a 2xx status. Run this check in a scheduled job rather than on every pre-commit hook to avoid network dependencies in local development.

CI/CD Integration

Manual one-off validation is insufficient for feeds that publish on daily or weekly schedules. Embed validate_agency_csv in your automation layer:

• Pre-commit hook — a lightweight column-presence and regex check (no network calls) that runs in under one second and blocks commits with obviously malformed agency.txt rows.
• Scheduled pipeline step — full Pydantic validation plus the requests.head() URL liveness check, triggered on every feed export. GitHub Actions, GitLab CI, and Airflow DAGs all support this pattern. See automating GTFS version control with Python scripts for a complete pipeline example including feed diffing and changelog generation.
• Threshold alerts — block feed publication if agency_id count changes unexpectedly (an operator was added or removed without a deliberate config change), or if any timezone or language code deviates from the approved allowlist for your region.

These guardrails catch drift before consumers ingest broken data, which is especially important when agency.txt changes propagate to GTFS-Realtime bridges: a mismatched agency_id between static and realtime feeds causes vehicle positions to detach from scheduled trips, producing blank maps and missing ETAs.

Related

• Agency Metadata and Feed Versioning Practices — parent page covering feed_info.txt, version hashing, and changelog generation
• Automating GTFS Version Control with Python Scripts — CI/CD pipeline that diffs consecutive feed snapshots and detects agency_id drift
• Converting Local Transit Times to UTC in Python — applying IANA zone rules correctly after agency_timezone is validated
• GTFS Validation Rules and Common Schema Errors — broader schema validation covering foreign-key integrity across all GTFS files
• GTFS Feed Architecture and Fundamentals — top-level guide to GTFS file relationships, data contracts, and pipeline architecture