Data Schema Validation Rules

In modern aviation operations, raw telemetry, maintenance manifests, and scheduling payloads arrive from heterogeneous sources at scale. Without deterministic validation, a single malformed record silently corrupts downstream pairing engines, triggers regulatory violations, and cascades into costly operational disruptions. Within the broader flight data ingestion architecture, schema validation is the primary compliance gatekeeper: it translates regulatory mandates into executable, auditable logic before any record reaches flight dispatch systems or crew scheduling platforms. Everything a duty time validation rule engine later computes inherits the correctness — or the defects — admitted at this boundary.

The Compliance Challenge This Layer Solves

The scoped problem is narrow but unforgiving: given a stream of operational records from feeds that disagree on format, precision, and vocabulary, decide — deterministically and at ingestion latency — which records are structurally sound, semantically meaningful, and legally usable, and quarantine the rest with enough context to remediate them. A validation layer that merely checks “is this valid JSON” is worthless here, because the failures that matter in aviation are not syntactic. They are an ICAO code that parses but names a decommissioned aerodrome, a duty segment whose block-in precedes its block-out, a timestamp that is well-formed but carries no zone, or an aircraft registration that is grammatically correct for one authority and illegal for another.

Three properties make this hard. First, correctness is layered: a payload can be perfectly typed and still be non-compliant, so validation must run in stages rather than as one pass. Second, semantic truth is external — whether an airport code is active, whether a crew member’s medical certificate is current, whether an aircraft is under an open airworthiness directive — which means validation depends on live registry lookups that can fail transiently. Third, the same record feeds regulatory arithmetic under both FAA and EASA regimes, so the schema must preserve every field those calculations need (acclimatisation context, report time, sector count) even when the source feed omits or mangles it. A validation layer that satisfies an authority inspection must model all three concerns as first-class stages, not as branches buried in a parsing loop.

Layered Architecture: Syntactic vs. Semantic Gates

Production-grade validation requires a strict separation between syntactic verification and semantic compliance. Syntactic checks enforce structural integrity — field presence, data typing, and hierarchical nesting — typically expressed as pydantic models or JSON Schema documents. Semantic validation operates at the domain layer, cross-referencing payloads against authoritative registries: ICAO airport codes must resolve to active aerodromes, aircraft type designators must match certified configurations, and timestamps must be timezone-aware ISO 8601 instants rather than ambiguous local strings.

When processing high-volume telemetry through flight log parsing pipelines, a two-stage gate proves operationally resilient. A lightweight schema check at the ingestion edge immediately rejects structurally invalid payloads, while a contextual validation pass verifies operational feasibility against live aircraft status, crew qualification matrices, and historical routing data. This pattern eliminates downstream reconciliation overhead and prevents scheduler fatigue caused by phantom assignments that were never legal to begin with.

Schema and Data Structure Design

The validation layer’s data model separates three concerns operators routinely conflate: the raw record as received, the versioned rules that judge it, and the structured outcome of that judgement. Keeping the ruleset in its own effective-dated table is what lets a regulatory revision be diffed, reviewed, and rolled back without touching a line of ingestion code — the same discipline the EASA FTL compliance frameworks apply to their FDP tables.

The core entities are the raw_payload as received from a source feed; the effective-dated validation_ruleset that supplies the active field constraints and semantic predicates keyed by revision date; the field_constraint rows that define type, bound, and cross-field rules; the validation_result that records pass/fail per record with its violation tier; the quarantine_record that holds rejected payloads with remediation metadata; and the authoritative reference registries — airport_registry, aircraft_type_registry, and crew_qualification — that the semantic gate resolves against. Every temporal column is stored in UTC with the originating zone retained as metadata, so local report time is derived at evaluation, never persisted ambiguously.

Field naming, unit standardisation, and audit lineage across the scheduling, payroll, and compliance modules all follow the shared crew duty time taxonomy mapping. Anchoring the schema to that taxonomy is what prevents the validator from misclassifying a positioning segment as a revenue sector, or a standby callout as the start of a flight duty period — the two most common sources of false violations at this boundary.

Regulatory Mapping and Temporal Constraints

Crew scheduling introduces distinct validation complexity because pairing legality depends on interdependent temporal constraints rather than isolated field values. Schema rules must model the relationships between flight segments, ground-handling windows, and mandatory rest periods, not just the shape of each field. The mapping below is the ground truth the validator encodes.

• FAA §117.13 — Flight duty period limits. The maximum FDP is a function of report time and number of flight segments; the schema must therefore preserve both, and reject any duty payload that omits acclimatisation context. The authoritative wording lives in the FAA Part 117 official text.
• FAA §117.23 — Cumulative limits. Caps flight duty period at 60 hours in any 168 consecutive hours and 190 hours in any 672 consecutive hours, and flight time at 100 hours in any 672 hours. These rolling windows are the reason the schema validates segment-level rows rather than daily totals — a validator that only checks a per-day field cannot see a 168-hour breach.
• FAA §117.25 — Rest period. Requires a minimum rest before an FDP relative to the preceding duty, so a rest payload is validated against the duty that precedes it, never in isolation.
• EASA ORO.FTL.205 / ORO.FTL.210 / ORO.FTL.235. For dual-jurisdiction carriers the same record must satisfy EASA’s daily FDP table, its 7/14/28-day cumulative caps, and its rest minima; the FAA Part 117 rule schema design and its EASA counterpart consume the fields this validator guarantees. The schema therefore tags every segment with its governing jurisdiction so the correct predicate set is selected.

Because both regimes key their arithmetic on report time and sector count, durations must be consistently represented in UTC or explicitly tagged with IANA time zones, and aircraft registration formats must conform to national authority standards (N-prefix for the FAA, G-prefix for the UK CAA). When these rules are enforced at ingestion, downstream engines receive only legally coherent, temporally unambiguous inputs — and the source records flow in through the crew roster API integration already normalised to this contract.

Python Implementation Walkthrough

Production implementations express the syntactic contract as typed models so a malformed payload fails at the boundary rather than deep inside a compliance calculation. Using pydantic compiles the schema into an optimised validator, keeping per-record latency low during high-throughput ingestion. The model below is the syntactic gate for a duty segment: it enforces field presence, numeric bounds, timezone-awareness, and cross-field ordering in one construction step.

from __future__ import annotations

from datetime import datetime, timezone
from enum import Enum

from pydantic import BaseModel, Field, field_validator, model_validator


class Jurisdiction(str, Enum):
    FAA = "faa"
    EASA = "easa"


class DutySegment(BaseModel):
    correlation_id: str = Field(..., min_length=1)
    crew_member_id: str = Field(..., min_length=1)
    origin: str = Field(..., min_length=3, max_length=4)
    destination: str = Field(..., min_length=3, max_length=4)
    block_out_utc: datetime
    block_in_utc: datetime
    sectors: int = Field(..., ge=1, le=8)
    jurisdiction: Jurisdiction

    @field_validator("block_out_utc", "block_in_utc")
    @classmethod
    def must_be_utc(cls, value: datetime) -> datetime:
        if value.tzinfo is None:
            raise ValueError("timestamps must be timezone-aware")
        return value.astimezone(timezone.utc)

    @model_validator(mode="after")
    def check_temporal_and_station(self) -> "DutySegment":
        if self.block_in_utc <= self.block_out_utc:
            raise ValueError("block_in must follow block_out")
        if self.origin == self.destination:
            raise ValueError("origin and destination must differ")
        return self

The syntactic gate guarantees shape; it cannot know whether origin names a live aerodrome or whether the crew member is qualified on the type. That is the semantic gate’s job, and keeping it a separate, pure function is what makes every verdict reproducible from the audit log months later.

from dataclasses import dataclass


@dataclass(frozen=True)
class Violation:
    field: str
    constraint: str
    tier: str          # "syntactic" | "semantic" | "regulatory"
    record_id: str
    remediation: str


def semantic_gate(segment: DutySegment, registries) -> list[Violation]:
    """Cross-reference a structurally valid segment against live registries."""
    problems: list[Violation] = []
    for field, code in (("origin", segment.origin), ("destination", segment.destination)):
        if not registries.airport_is_active(code):
            problems.append(
                Violation(
                    field=field,
                    constraint="airport must be an active ICAO/IATA aerodrome",
                    tier="semantic",
                    record_id=segment.correlation_id,
                    remediation=f"verify {code} against the current airport registry",
                )
            )
    if not registries.crew_qualified(segment.crew_member_id, segment.origin):
        problems.append(
            Violation(
                field="crew_member_id",
                constraint="crew must hold a current qualification for the assignment",
                tier="semantic",
                record_id=segment.correlation_id,
                remediation="refresh qualification matrix or reassign",
            )
        )
    return problems

Structured violations — never unstructured log strings — are what let compliance dashboards trace non-compliance to its origin. Each carries the offending field, the violated constraint, the tier, the source record identifier, and a remediation directive, so a data steward can act without re-deriving what went wrong.

Rolling Window and Temporal Aggregation Patterns

Field-level validation is stateless, but the regulatory caps are not: §117.23 requires rolling totals over 168- and 672-hour windows, and the EASA equivalents over 7, 14, and 28 days. A validator that ignores these admits records that are individually legal yet collectively illegal. Timezone drift is the main source of cumulative error, so every input is stamped in ISO 8601 UTC before it enters the aggregation layer. Where the data lands in PostgreSQL, frame-bounded window functions express the caps directly; see the PostgreSQL window functions reference for the frame semantics.

-- Rolling 168-hour flight-duty-period total per crew member (FAA §117.23),
-- evaluated at each duty start so an over-cap segment is flagged at ingestion.
SELECT
    crew_member_id,
    block_out_utc,
    SUM(fdp_minutes) OVER (
        PARTITION BY crew_member_id
        ORDER BY block_out_utc
        RANGE BETWEEN INTERVAL '168 hours' PRECEDING AND CURRENT ROW
    ) AS rolling_fdp_minutes_168h
FROM duty_segment
ORDER BY crew_member_id, block_out_utc;

For pre-admission validation inside the ingestion worker, the same logic runs in memory over a Polars or Pandas frame, where a rolling group-by keyed on the duty timestamp reproduces the SQL frame without a database round trip. Whichever engine is used, the window boundaries must be inclusive of the exact regulatory span — an off-by-one on the 168-hour boundary silently under-counts duty and lets an illegal record through the gate it was meant to catch.

Integration Points

This validation layer is one stage in a longer chain and depends on clean contracts with its neighbours:

• Upstream — flight log parsing pipelines. Deliver raw records already stripped of proprietary formatting and reconciled across feeds; the validator assumes one canonical candidate per physical flight, not three conflicting ones.
• Vocabulary — crew duty time taxonomy mapping. Supplies the canonical field names and unit conventions so sectors, positioning, standby, and rest are classified identically before they are validated or counted.
• Downstream — duty time validation rule engines. Consume only records that clear both gates, so the engines never spend cycles defending against malformed input and can assume every field they read is present, typed, and UTC-normalised.
• Scale — async batch processing workflows. Validation must be decoupled from network I/O to avoid event-loop starvation; running the semantic gate’s registry lookups in worker pools or via asyncio.gather with bounded concurrency keeps throughput scaling with available compute during irregular-operations surges.
• Perimeter — system security and access boundaries. Every validation outcome is written to an append-only ledger under role-based access control, so who overrode a quarantine and when is always attributable.

When validation depends on external registry lookups — verifying crew medical certificates or open airworthiness directives — transient network failures require disciplined retry logic. Exponential backoff with jitter, circuit breakers, and idempotent request signatures preserve resilience without breaching SLA thresholds. Records whose semantic checks cannot be completed are routed to a dead-letter queue for review rather than admitted or dropped, keeping the primary stream clean while maintaining a complete audit trail.

Testing and Edge Cases

Because validation is time-sensitive and registry-dependent, example-based tests miss the cases that matter. Property-based testing with hypothesis generates thousands of synthetic payloads and asserts invariants — for instance, that no segment with block_in <= block_out is ever accepted, and that UTC conversion is round-trip stable. The datetime and zoneinfo handling that underpins these tests is documented in the Python datetime documentation.

from datetime import timedelta
from zoneinfo import ZoneInfo

from hypothesis import given
from hypothesis import strategies as st


@given(
    block_out=st.datetimes(timezones=st.just(ZoneInfo("UTC"))),
    duration=st.integers(min_value=1, max_value=18 * 60),
)
def test_valid_segment_is_temporally_ordered(block_out, duration):
    segment = DutySegment(
        correlation_id="c-1",
        crew_member_id="crew-1",
        origin="KJFK",
        destination="EGLL",
        block_out_utc=block_out,
        block_in_utc=block_out + timedelta(minutes=duration),
        sectors=1,
        jurisdiction=Jurisdiction.FAA,
    )
    assert segment.block_in_utc > segment.block_out_utc

The boundary conditions that most often break aviation validators are specific and must each become a named, manually verified test case:

• Naive timestamps. A well-formed local string with no zone must be rejected, not silently coerced to server time — the single highest-value guarantee this layer provides.
• Daylight-saving transitions. A duty spanning a spring-forward or fall-back change must convert through zoneinfo, not a fixed offset, or its FDP band shifts and a legal duty is flagged (or an illegal one cleared).
• Rolling-window edges. A segment landing exactly on a 168-hour or 28-day boundary must be counted inclusively; the property test should assert the cumulative total is invariant to whether the boundary record is expressed in local or UTC time.
• Registry staleness. A payload validated against a cached registry snapshot older than its refresh interval must be re-checked, so a code that was active yesterday but decommissioned today does not clear the semantic gate.

Predicate logic is validated against the current published regulatory constants before every deployment, with the ruleset version-controlled so a rule revision is a reviewable diff rather than a code change.

Explore This Topic in Depth

• Validating IATA SSIM files with Pydantic — mapping complex seasonal schedule structures to strict, type-safe models and streaming validation over generators so multi-gigabyte SSIM manifests never load entirely into RAM.

Related

• Flight log parsing pipelines — the upstream stage that hands this layer reconciled, canonical records.
• Crew roster API integration — how roster payloads reach the validation boundary already normalised.
• Async batch processing workflows — decoupling validation from I/O to scale during IROPS surges.
• Crew duty time taxonomy mapping — the shared vocabulary this layer classifies against.
• Duty time validation rule engines — the downstream consumers of every record that clears both gates.

Back to Flight Data Ingestion & System Sync.