Webhook Event Delivery for DVIR Pipelines

When a DVIR is finalized and the defect classification engine has assigned it a severity score, the compliance decision has to travel — out of the pipeline and into the telematics, maintenance, and driver-facing systems that enforce it. Webhooks are the transport for that journey, and they are the point where a durable internal decision meets an unreliable external network. A webhook that drops an out-of-service hold event does not merely lose data; it leaves an unsafe vehicle dispatched against a clean-looking record, which is exactly the failure 49 CFR § 396.9 and § 396.11 exist to prevent. This guide specifies the general outbound webhook delivery layer for the telematics and platform integration section: a transactional outbox, a dispatcher that signs and retries, and a dead-letter path that guarantees no safety-critical event is silently lost.

The delivery layer is deliberately vendor-neutral. It carries the canonical DvirComplianceEvent defined in the platform integration architecture and knows nothing about Samsara payload shapes or Geotab authentication — those live in the vendor adapters. What this layer owns is the delivery guarantee: at-least-once, idempotent, cryptographically signed, and auditable. Get that contract right once and every destination inherits it, from the work-order sync that opens a CMMS record to the ELD identity linkage that binds the event to a driver.

The delivery layer is a background service that reads an outbox and writes to remote endpoints. It never runs inside a request handler, because delivery latency and retries must not block the pipeline that produced the event. Target Python 3.10+ (the code uses match/case and X | Y unions) with:

• Pydantic 2.x — the DvirComplianceEvent contract inherited from the platform integration architecture; the delivery layer treats it as opaque signed bytes.
• httpx — an async HTTP client with per-request timeouts and connection pooling for the outbound POSTs.
• SQLAlchemy 2.x over PostgreSQL — the outbox table, read with SELECT ... FOR UPDATE SKIP LOCKED so multiple dispatcher workers never claim the same row.
• hmac / hashlib / secrets (stdlib) — HMAC-SHA256 request signing and constant-time comparison.
• Celery 5.x or a custom asyncio worker — scheduling the dispatcher poll loop and the backoff-delayed retries.

The event contract is fixed upstream. This layer must not add, drop, or reorder fields, because the audit_hash and the HMAC signature are both computed over the canonical serialization — any mutation here would invalidate the signature the receiver checks. Treat the payload bytes as immutable from the moment they land in the outbox.

Two records govern delivery: the outbox row that holds the event and its delivery state, and the signed request envelope that goes on the wire. The outbox row is the durable source of truth for what must be delivered and how far each delivery has progressed.

The status enum is a small state machine: PENDING → INFLIGHT → DELIVERED, or INFLIGHT → PENDING on a retryable failure (with attempts incremented and next_attempt_at pushed forward), or INFLIGHT → DEAD_LETTER when the budget is exhausted or the error is non-retryable. Encode it as a frozen Pydantic model so an invalid transition is a validation error rather than a silent corruption:

from datetime import datetime
from typing import Literal
from uuid import UUID

from pydantic import BaseModel, Field

DeliveryStatus = Literal["PENDING", "INFLIGHT", "DELIVERED", "DEAD_LETTER"]
Destination = Literal["telematics", "cmms", "eld"]


class OutboxRow(BaseModel):
    model_config = {"frozen": True}

    event_id: UUID
    payload: bytes                       # canonical signed bytes, never mutated
    audit_hash: str
    destination: Destination
    status: DeliveryStatus = "PENDING"
    attempts: int = Field(default=0, ge=0)
    next_attempt_at: datetime
    last_status_code: int | None = None

Delivery runs as an ordered loop: claim due rows, sign, POST, then persist the outcome. Each step is written so that a crash at any point leaves the outbox in a state a later poll can safely resume — the delivery is at-least-once by construction, and correctness depends on the receiver being idempotent, which is specified in idempotent webhook retry and signature verification.

1. Claim due events. Select PENDING rows whose next_attempt_at is in the past, using FOR UPDATE SKIP LOCKED so concurrent workers partition the work without contending. Mark each claimed row INFLIGHT.
2. Sign the payload. Compute an HMAC-SHA256 over the exact outbox payload bytes plus a signed timestamp, and attach it as a header alongside the event_id idempotency key. Never re-serialize the event — sign the bytes as stored.
3. POST with a bounded timeout. Issue the HTTP POST through httpx with a strict connect and read timeout. A hung endpoint must fail fast into the retry path rather than tie up a worker.
4. Classify the outcome. A 2xx marks the row DELIVERED. A 5xx, 429, or timeout increments attempts and reschedules with exponential backoff and jitter. A 4xx client error (400, 401, 403, 422) is non-retryable and dead-letters immediately.
5. Record the attempt. Append the attempt — event_id, destination, status code, attempt number, timestamp, audit_hash — to the append-only delivery log before releasing the row, so no delivery outcome is unlogged.

The dispatcher’s signing and outcome classification are the load-bearing pieces. Sign over stored bytes, and grade the HTTP result deterministically:

import hmac
import hashlib
import random
from datetime import datetime, timezone


def sign_request(payload: bytes, secret: bytes, ts: int) -> str:
    """HMAC-SHA256 over a timestamped payload. The receiver recomputes this
    over the raw body it received and compares in constant time."""
    mac = hmac.new(secret, f"{ts}.".encode() + payload, hashlib.sha256)
    return f"t={ts},v1={mac.hexdigest()}"


def backoff_delay(attempts: int, base: float = 2.0, cap: float = 900.0) -> float:
    """Exponential backoff with full jitter, capped at 15 minutes.
    Jitter spreads a reconnect storm so retries do not synchronize."""
    ceiling = min(cap, base * (2 ** attempts))
    return random.uniform(0, ceiling)


def classify(status_code: int | None, attempts: int, max_attempts: int) -> str:
    match status_code:
        case s if s is not None and 200 <= s < 300:
            return "DELIVERED"
        case 400 | 401 | 403 | 422:
            return "DEAD_LETTER"                 # non-retryable client error
        case _ if attempts + 1 >= max_attempts:
            return "DEAD_LETTER"                 # retry budget exhausted
        case _:
            return "RETRY"                       # 5xx, 429, timeout, network

import httpx


async def deliver(row: OutboxRow, endpoint: str, secret: bytes,
                  max_attempts: int = 8) -> str:
    """Deliver one event at-least-once. Returns the resulting status.
    A safety-critical OOS event is never dropped: it retries or dead-letters."""
    ts = int(datetime.now(timezone.utc).timestamp())
    headers = {
        "Content-Type": "application/json",
        "Idempotency-Key": str(row.event_id),
        "X-DVIR-Signature": sign_request(row.payload, secret, ts),
    }
    try:
        async with httpx.AsyncClient(timeout=httpx.Timeout(5.0, read=10.0)) as client:
            resp = await client.post(endpoint, content=row.payload, headers=headers)
        outcome = classify(resp.status_code, row.attempts, max_attempts)
        record_attempt(row, resp.status_code)      # append-only delivery log
    except (httpx.TimeoutException, httpx.TransportError):
        outcome = classify(None, row.attempts, max_attempts)
        record_attempt(row, None)
    return outcome

Not every event carries the same urgency, and the delivery layer grades its behavior by the compliance_action the event carries. An OOS_HOLD — produced when severity reaches the critical band of 70–100 or a defect meets the DOT Out-of-Service Criteria under § 396.9 — is the event that must never be lost. It gets the full retry budget, and if it exhausts that budget it dead-letters with an immediate page to on-call, because an undelivered out-of-service hold means a vehicle the carrier believes is held may still be dispatched. The out-of-service determination itself is made upstream in out-of-service criteria and OOS hold triggers; this layer’s obligation is to carry that determination to the enforcing platform without loss.

A REPAIR_QUEUE event (major, 35–69) attaches the regulated 24-hour repair window obligation, so its delivery is durable but its dead-letter alert can wait for business-hours triage. A SCHEDULED_PM event (minor, 0–34) is the lowest priority and may batch. Encoding the priority into the delivery policy — retry aggressiveness, alert routing, and dead-letter escalation — is what keeps a flood of low-severity preventive-maintenance events from crowding out the one OOS hold that actually grounds a truck. The dead-letter machinery is shared with the ingestion side’s exception handling and dead-letter routing, and permanently failed deliveries are triaged there.

Priority also governs ordering. Within a single destination, an OOS_HOLD must not sit behind a backlog of SCHEDULED_PM events waiting to drain, so the dispatcher claims due rows ordered by an action-derived priority column, not by insertion time alone. The delivery layer does not, however, guarantee a strict global order across events — an event that lifts a hold could in principle overtake the event that set it if both are in flight at once. Guard against that inversion at the source: emit the clearing event only after the setting event has reached a DELIVERED state for that destination, or carry a monotonically increasing version on the event so a receiver can reject an out-of-order clearance whose version predates the hold it would lift. The platform integration architecture stamps that version on every DvirComplianceEvent for exactly this reason.

In production the dispatcher runs as a horizontally scaled pool, and three properties keep it correct under load. Idempotent claiming — FOR UPDATE SKIP LOCKED guarantees two workers never deliver the same row, and the event_id idempotency key guarantees the receiver collapses any duplicate that a retry produces anyway. Cryptographic chaining — each delivery-log entry hashes the prior entry together with the attempt, so a deleted or reordered attempt is detectable during an audit, the same tamper-evident construction used in audit logging and tamper-evident event chains. Per-endpoint delivery-status tracking — each destination advances independently, so a Samsara outage never stalls delivery to the CMMS.

The vendor-specific translation happens at the edge of this layer, not inside it. The dispatcher hands signed canonical bytes to an adapter, and the adapter maps them to a Samsara, Geotab, or Motive request — the concrete mappings are in syncing DVIR severity scores to Samsara and Geotab via webhook and the vendor-neutral adapter layer. Keeping translation out of the delivery core means one signing routine, one retry policy, and one audit format serve every vendor, and adding a new destination is a new adapter rather than a change to the guarantee.

Observability closes the loop. Because the outbox row carries status, attempts, and last_status_code, the delivery backlog is a queryable table rather than a black box: an operator can ask how many OOS_HOLD events are PENDING past their next_attempt_at, or which destination has the deepest retry backlog, and alert on either before a driver notices a stale board. Export those counts as metrics — pending depth per destination, attempts-per-event percentiles, dead-letter rate graded by compliance_action — so a degrading downstream platform is visible as a rising retry curve well before any event exhausts its budget. When a destination has been failing long enough that its backlog threatens the retry cap, pause new deliveries to it and let the outbox hold the events durably rather than burning every event’s budget against an endpoint that is already known to be down.

• Transactional outbox — write the event in the same transaction that finalizes the DVIR; never emit from a request handler.
• At-least-once, never at-most-once — retry every retryable failure; a dropped OOS hold is a safety incident, a duplicate is a harmless no-op.
• Sign stored bytes — HMAC-SHA256 over the exact outbox payload; never re-serialize before signing.
• Bounded backoff with jitter — exponential backoff capped at 15 minutes with full jitter so retries never synchronize into a storm.
• Non-retryable errors dead-letter fast — a 400 or 401 burns no retry budget; fix the credential or mapping instead.
• Grade by compliance_action — an OOS_HOLD dead-letter pages immediately; a SCHEDULED_PM waits for a business-hours sweep.
• Log every attempt immutably — chain each delivery-log entry by SHA-256 so the delivery history is tamper-evident.

• Telematics, ELD & Maintenance Platform Integration — the parent architecture and the canonical event this layer carries.
• Syncing DVIR Severity Scores to Samsara and Geotab via Webhook — the concrete vendor payload mapping over this transport.
• Idempotent Webhook Retry and Signature Verification — the receiver-side dedup and signature checks this layer depends on.
• Samsara, Geotab and Motive Platform Adapters — the vendor translation that sits at the edge of this delivery core.
• Out-of-Service Criteria and OOS Hold Triggers — where the OOS_HOLD action this layer must never drop is determined.

Back to Telematics, ELD & Maintenance Platform Integration.