Exception Handling and Dead-Letter Routing

An ingestion pipeline that drops a Driver Vehicle Inspection Report when a parser throws is not merely buggy — it has manufactured a compliance gap. Under 49 CFR § 396.3© the carrier must retain the record of every inspection and the maintenance action it triggers, so a DVIR that vanishes because a malformed VIN raised a ValidationError is a records-retention failure that no amount of downstream correctness can undo. The job of the exception layer inside the DVIR Ingestion & Digital/Paper Parsing Workflows architecture is therefore blunt: no inspection payload may ever leave the system without either becoming a valid record or landing, intact and inspectable, in a place a human is obligated to review. Silent loss is not an available outcome.

The engineering discipline that guarantees this is a strict separation of transient failures from permanent ones. A transient failure — a database deadlock, a timed-out OCR worker, a throttled telematics API — will likely succeed on a later attempt, so retry it with exponential backoff and a hard cap. A permanent failure — a payload that fails schema validation, an unresolvable defect code, a corrupt PDF — will fail identically forever, so retrying it only burns compute and delays the human decision it actually needs; route it to a dead-letter queue with enough context to diagnose and replay it. This page specifies that classifier, the failure envelope that preserves the original bytes for audit, the alerting that makes a growing dead-letter backlog impossible to ignore, and the replay path that re-enqueues a payload once the underlying defect is fixed. The concrete queue implementation — the failure envelope, the round-trip test, and idempotent replay — is built out in Designing a Dead-Letter Queue for Failed DVIR Payloads.

Target Python 3.10+ — the classifier below uses StrEnum, structural match/case, and X | Y unions. The exception layer sits between the ingestion channels and the canonical record store, so it depends on the same contracts as the rest of the pipeline rather than inventing its own. The stack is:

• pydantic (v2) — model the IngestionFailure envelope and validate every payload before it is admitted; a ValidationError here is the canonical permanent failure signal.
• tenacity — declarative retry with exponential backoff and jitter, so transient-failure retry policy is code you can read and test rather than hand-rolled sleep loops.
• A durable broker — Redis Streams or Amazon SQS backs both the work queue and the dead-letter queue. The queue mechanics are the subject of the dead-letter queue design guide; this page is the policy layer above it.
• httpx — outbound calls to telematics and CMMS platforms whose timeouts and 5xx responses are the most common transient failures.

The exception layer does not define the record schema. A payload is permanently failed the moment it cannot satisfy the canonical contract from Standardized DVIR JSON Schema Design, and the original bytes it preserves are hashed and retained under the same rules the Immutable Audit Trail and WORM Retention guide specifies for admitted records. Failures produced during high-volume bursts from the Async Batching for High-Volume Ingestion stage flow through exactly this classifier before anything is retried or dead-lettered.

Every failure is normalized into a single IngestionFailure envelope. The envelope is not a log line — it is a durable, replayable record that preserves the original payload by reference, captures where and why processing stopped, and tracks how many times it has been attempted. It is the only artifact a reviewer or an auditor needs to understand and recover a failed inspection.

from datetime import datetime
from enum import StrEnum
from pydantic import BaseModel, Field

MAX_RETRIES = 5  # hard cap; beyond this a transient failure is treated as permanent

class Stage(StrEnum):
    INGEST = "ingest"
    VALIDATE = "validate"
    PARSE = "parse"
    NORMALIZE = "normalize"
    DISPATCH = "dispatch"

class ErrorClass(StrEnum):
    TRANSIENT = "transient"   # retry with backoff
    PERMANENT = "permanent"   # route to dead-letter for human review

class Disposition(StrEnum):
    RETRYING = "retrying"
    DEAD_LETTER = "dead_letter"
    REPLAYED = "replayed"
    RESOLVED = "resolved"

class IngestionFailure(BaseModel):
    payload_ref: str = Field(pattern=r"^[0-9a-f]{64}$")  # SHA-256 of the original bytes
    stage: Stage
    error_class: ErrorClass
    error_code: str
    attempt: int = Field(ge=1)
    first_seen: datetime
    last_error: str
    disposition: Disposition

The payload_ref is a content hash, not a copy of the payload inside the envelope. Store the original bytes once in a write-once object store keyed by that hash, and the envelope stays small, deduplicates re-uploads of the same failing DVIR, and still lets an auditor retrieve the exact bytes that failed. Never inline the raw payload into the envelope that fans out to logs and alerts — that is how driver PII leaks into systems that were never scoped to hold it.

The exception layer is a state machine with one rule: a payload is either progressing toward a valid record, waiting on a bounded retry, or sitting in the dead-letter queue awaiting a human. It is never simply gone.

1. Classify the failure. Map the caught exception to transient or permanent from an explicit table, not from a bare except Exception. Anything you cannot positively classify as transient is permanent — default to the dead-letter queue, because retrying an unclassifiable error risks an infinite loop, while dead-lettering it only costs one human review.

import httpx
from pydantic import ValidationError

# Positive list of transient exception types. Everything else is permanent.
TRANSIENT_TYPES: tuple[type[Exception], ...] = (
    httpx.TimeoutException,
    httpx.ConnectError,
    ConnectionResetError,
    TimeoutError,
)

def classify(exc: Exception) -> ErrorClass:
    match exc:
        case ValidationError() | ValueError() | KeyError():
            return ErrorClass.PERMANENT     # bad data fails identically forever
        case httpx.HTTPStatusError() as e:
            # 429 and 5xx are transient; 4xx (except 429) are a permanent client fault.
            code = e.response.status_code
            transient = code == 429 or 500 <= code < 600
            return ErrorClass.TRANSIENT if transient else ErrorClass.PERMANENT
        case _ if isinstance(exc, TRANSIENT_TYPES):
            return ErrorClass.TRANSIENT
        case _:
            return ErrorClass.PERMANENT     # unknown -> dead-letter, never retry blindly

2. Retry transient failures with capped exponential backoff. A transient failure gets a bounded number of attempts with exponentially increasing, jittered delays so a downstream outage does not turn into a retry storm the instant it recovers. Cap the attempts: once attempt exceeds MAX_RETRIES, reclassify the payload as permanent and dead-letter it. An unbounded retry is indistinguishable from silent loss — the record never lands anywhere a human looks.

from tenacity import (
    retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type,
)

@retry(
    retry=retry_if_exception_type(TRANSIENT_TYPES),
    wait=wait_exponential_jitter(initial=1, max=60),  # 1s, 2s, 4s ... capped, jittered
    stop=stop_after_attempt(MAX_RETRIES),
    reraise=True,   # after the cap, re-raise so the caller can dead-letter
)
def process_with_retry(payload: bytes) -> dict:
    return normalize_dvir(payload)  # raises ValidationError (permanent) or httpx errors (transient)

3. Route permanent failures to the dead-letter queue. Wrap the payload reference and error context in an IngestionFailure, persist it, and stop. Do not delete the source payload, do not emit a partial record, and do not swallow the exception. The envelope’s disposition is dead_letter and its first_seen starts the backlog-age clock that alerting watches.

import hashlib
from datetime import datetime, timezone

def to_dead_letter(payload: bytes, stage: Stage, exc: Exception, attempt: int) -> IngestionFailure:
    return IngestionFailure(
        payload_ref=hashlib.sha256(payload).hexdigest(),  # bytes stored once, keyed by hash
        stage=stage,
        error_class=ErrorClass.PERMANENT,
        error_code=f"{stage}.{type(exc).__name__}",
        attempt=attempt,
        first_seen=datetime.now(timezone.utc),
        last_error=str(exc)[:500],   # summary only — never the raw payload
        disposition=Disposition.DEAD_LETTER,
    )

4. Replay after the fix. When a reviewer or a code fix resolves the root cause — a corrected layout map, a widened schema enum, a recovered downstream — re-enqueue the preserved bytes through the same processing path. Replay must be idempotent: re-processing a payload that partially succeeded before must not create a second compliance record. Key the record write on the payload_ref so a replay of an already-admitted inspection resolves to the existing record instead of duplicating it. The end-to-end round-trip and idempotent replay are implemented and tested in Designing a Dead-Letter Queue for Failed DVIR Payloads.

The classifier’s output maps directly to an obligation, not a preference. Treat the routing table below as a compliance contract: each failure state has exactly one correct destination, and “drop it” is never a cell in the table.

A poison message — a payload that does not merely fail validation but crashes the worker process (an unparseable PDF that segfaults a native OCR binary, a decompression bomb) — is the most dangerous case, because a naive broker will redeliver it to the next worker and take the whole pool down. Detect it by tracking delivery count on the message itself and quarantine to the dead-letter queue after the first crash rather than the fifth. A DVIR that crashes a worker is still a DVIR the carrier must retain; quarantine preserves it for review without letting it deny service to every other inspection in the queue.

The retention obligation outlives the failure. A dead-lettered DVIR is a record within the meaning of § 396.3© — its original bytes must be retained and hash-anchored exactly as an admitted record would be, and its lifecycle (dead_letter → replayed → resolved) must be captured in the tamper-evident event chain described in Audit Logging and Tamper-Evident Event Chains. If an auditor asks what happened to an inspection that a driver swears they submitted, the answer must be a durable envelope, not a shrug.

The exception layer is a shared service, not a per-consumer afterthought. Every ingestion channel — the OCR pipeline, the mobile export endpoint, the field-mapping normalizer — hands its failures to the same classifier and the same dead-letter queue, so there is one place to watch a backlog, one alerting policy, and one replay tool. Emit a structured event on every disposition change (retrying, dead_letter, replayed, resolved) onto the audit stream so the backlog is observable in real time rather than discovered during an audit.

Alerting must be driven by the two signals that predict compliance risk: backlog age and error-code concentration. A single dead-lettered DVIR is routine; a dead-letter queue whose oldest envelope is four hours old, or a sudden spike of one error_code (a carrier changed its form layout and every VIN crop now fails), is an incident. Page on both. Because outbound platform calls to telematics and CMMS systems are the dominant source of transient failures, coordinate this layer’s retry policy with the delivery-side retries in Idempotent Webhook Retry and Signature Verification so the same event is not retried twice at two layers and double-fires an out-of-service hold. Deduplicate every retry and every replay on the payload_ref content hash; the severity bands that any recovered defect maps to are unchanged from the rest of this section: 0–34 minor, 35–69 major, 70–100 critical, with any DOT_OOS_CRITERIA defect held at the immovable critical floor of 70.

• No silent drops: every failure produces a durable IngestionFailure envelope; there is no code path that catches an exception and discards the payload.
• Positive transient classification: only exceptions on an explicit allow-list are retried; everything unclassified is dead-lettered, never retried blindly.
• Bounded retries: transient retries use capped exponential backoff with jitter and a hard MAX_RETRIES; exhaustion reclassifies to permanent.
• Poison-message quarantine: a payload that crashes the worker is quarantined after the first crash on its delivery count, protecting the worker pool.
• Bytes preserved by reference: the original payload is stored once, keyed by SHA-256, and referenced from the envelope — never inlined into logs or alerts where PII would leak.
• Idempotent replay: re-enqueuing a payload keys the record write on payload_ref, so replay cannot create a duplicate compliance record.
• Backlog alerting: dead-letter backlog age and per-error_code spikes both page; a growing queue is an incident, not a metric nobody reads.
• Audit-linked lifecycle: every disposition change is written to the tamper-evident event chain to satisfy § 396.3© retention.

• Designing a Dead-Letter Queue for Failed DVIR Payloads — the concrete Redis Streams or SQS queue, failure envelope, and idempotent replay tooling this policy layer sits on top of.
• Async Batching for High-Volume Ingestion — the burst-processing stage whose failures flow through this classifier.
• Idempotent Webhook Retry and Signature Verification — the delivery-side retry policy this layer coordinates with to avoid double-firing.
• Audit Logging and Tamper-Evident Event Chains — where the dead-letter lifecycle is recorded for § 396.3© retention.
• Standardized DVIR JSON Schema Design — the canonical contract whose validation failure is the definitive permanent-failure signal.

Back to DVIR Ingestion & Digital/Paper Parsing Workflows.