Calibrating Severity Thresholds from Historical Data

A severity threshold is only as good as the outcomes it predicts. If the minor/major boundary sits too high, defects that later became roadside violations or out-of-service (OOS) orders were routed to a low-priority preventive-maintenance queue and the vehicle kept running with a maturing problem; if it sits too low, mechanics drown in false alarms and the queue that matters gets ignored. This page fits that boundary from labeled history — which defects actually became OOS or were written up at a roadside inspection — and evaluates the fit with precision and recall against the known OOS labels. It supplies the empirical cut points consumed by Dynamic Threshold Tuning for Fleet Types, and the scores it calibrates against are the 0–100 values produced by the Severity Scoring Algorithms for DVIR Defects engine.

One rule overrides every statistic on this page: calibration may move the carrier-discretion minor/major boundary, but it must never lower the federal critical floor of 70 for any defect tagged DOT_OOS_CRITERIA. A model that fits history perfectly and drops an OOS-criteria brake defect below 70 is not a better model — it is a compliance violation under 49 CFR § 396.11©(2). The guardrail is enforced in code and rejects any calibrated floor that would breach it, no matter what the F-score says.

• Python 3.10+ with pandas 2.x + NumPy for the labeled dataset and the threshold sweep, and hashlib / datetime (stdlib) for the versioned, hashed output.
• A labeled historical dataset — each row carries a computed severity_score and a ground-truth became_oos label (whether the defect led to an OOS order or roadside violation), joined from inspection outcomes.
• The canonical severity model — bands are 0–34 minor, 35–69 major, 70–100 critical, with a DOT_OOS_CRITERIA critical floor of 70 that calibration cannot move.
• The calibration output contract — the fitted boundary is emitted as a versioned configuration the tuning layer reads, stamped with version, effective_date, and a SHA-256 audit_hash.

Because missing a real OOS defect is far worse than raising a false alarm, the sweep optimizes an F-beta score with beta > 1 to weight recall over precision. A false negative here is a truck on the road with an OOS condition; a false positive is a mechanic double-checking a healthy component.

Join each historical defect’s severity_score to its ground-truth outcome. The label must come from what actually happened downstream — an OOS order, a roadside Vehicle Maintenance violation — not from the tier the model assigned at the time, or the calibration learns its own past output instead of reality.

import pandas as pd

# Each row: one historically scored defect with its true downstream outcome.
# became_oos is the label: it led to an OOS order or a roadside violation.
def load_labeled_history(path: str) -> pd.DataFrame:
    df = pd.read_parquet(path)
    required = {"severity_score", "became_oos", "is_oos_criteria", "fleet_type"}
    missing = required - set(df.columns)
    if missing:
        raise ValueError(f"labeled history missing columns: {sorted(missing)}")
    # Drop rows whose label was derived from the model's own prior decision.
    df = df[df["label_source"] != "model_routing"].copy()
    return df.reset_index(drop=True)

Evaluate a candidate boundary by treating “score at or above the boundary” as the positive prediction and became_oos as the truth. Compute precision, recall, and an F-beta with beta = 2.0 so recall counts four times as much as precision in the harmonic mean.

def fbeta_at_threshold(
    scores: pd.Series, labels: pd.Series, threshold: float, beta: float = 2.0,
) -> dict:
    """Precision, recall, and F-beta for a candidate minor/major boundary."""
    predicted_positive = scores >= threshold
    tp = int((predicted_positive & labels).sum())
    fp = int((predicted_positive & ~labels).sum())
    fn = int((~predicted_positive & labels).sum())

    precision = tp / (tp + fp) if (tp + fp) else 0.0
    recall = tp / (tp + fn) if (tp + fn) else 0.0
    b2 = beta * beta
    denom = (b2 * precision) + recall
    fbeta = ((1 + b2) * precision * recall / denom) if denom else 0.0
    return {"threshold": threshold, "precision": precision,
            "recall": recall, "fbeta": fbeta}

Search only the carrier-discretion range. The sweep never proposes a boundary at or above 70, because the major/critical line is federally fixed; it tunes only where minor becomes major. Evaluate each candidate and keep the one with the highest F-beta, breaking ties toward the lower threshold so recall wins.

import numpy as np

CRITICAL_FLOOR = 70.0  # 0-100 scale; immovable for DOT_OOS_CRITERIA defects


def sweep_minor_major_boundary(
    df: pd.DataFrame, beta: float = 2.0,
) -> dict:
    """Deterministic sweep over the minor/major boundary in [1, 69]."""
    scores, labels = df["severity_score"], df["became_oos"].astype(bool)
    candidates = np.arange(1.0, CRITICAL_FLOOR, 1.0)  # never reaches 70
    results = [fbeta_at_threshold(scores, labels, t, beta) for t in candidates]
    # Highest F-beta wins; ties resolve to the lower threshold (favor recall).
    best = max(results, key=lambda r: (r["fbeta"], -r["threshold"]))
    return best

Overlay the guardrail last, as an imperative hard stop. Reject any calibrated boundary that reaches the critical floor, and independently assert that every DOT_OOS_CRITERIA defect in the training data still scores at or above 70 under the fitted configuration. If either check fails, keep the prior version and log the offending values rather than promote a calibration that erodes the floor.

from datetime import datetime, timezone
import hashlib
import json


def enforce_critical_floor(boundary: float, df: pd.DataFrame) -> float:
    """Reject any calibration that breaches the federal OOS floor. 396.11(c)(2)."""
    if boundary >= CRITICAL_FLOOR:
        raise ValueError(
            f"minor/major boundary {boundary} must stay below the "
            f"critical floor {CRITICAL_FLOOR}"
        )
    oos = df[df["is_oos_criteria"]]
    if not oos.empty and (oos["severity_score"] < CRITICAL_FLOOR).any():
        raise ValueError("a DOT_OOS_CRITERIA defect scores below the critical floor")
    return boundary


def emit_calibration(fleet_type: str, boundary: float, version: int) -> dict:
    """Serialize the fitted boundary with a hash for audit replay."""
    payload = {
        "fleet_type": fleet_type,
        "minor_major_boundary": boundary,
        "critical_floor": CRITICAL_FLOOR,  # always 70; recorded, never fitted
        "version": version,
        "effective_date": datetime.now(timezone.utc).isoformat(),
    }
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    payload["audit_hash"] = hashlib.sha256(canonical.encode()).hexdigest()
    return payload

Two properties matter most: the guardrail rejects any calibrated floor below 70 for an OOS-criteria defect, and the sweep is deterministic — the same labeled input always yields the same boundary, so a past calibration can be reproduced during an audit.

import pytest


def _frame(rows: list[dict]) -> pd.DataFrame:
    df = pd.DataFrame(rows)
    df["label_source"] = "inspection_outcome"
    return df


def test_guardrail_rejects_floor_below_70_for_oos_defect():
    df = _frame([
        {"severity_score": 62, "became_oos": True,
         "is_oos_criteria": True, "fleet_type": "long_haul_tractor"},
    ])
    # An OOS-criteria defect scored below 70 must fail calibration outright.
    with pytest.raises(ValueError):
        enforce_critical_floor(45.0, df)


def test_boundary_never_reaches_critical_floor():
    df = _frame([
        {"severity_score": s, "became_oos": s >= 50,
         "is_oos_criteria": False, "fleet_type": "regional_step_van"}
        for s in range(10, 69)
    ])
    best = sweep_minor_major_boundary(df)
    assert best["threshold"] < CRITICAL_FLOOR


def test_sweep_is_deterministic():
    rows = [
        {"severity_score": s, "became_oos": s >= 45,
         "is_oos_criteria": False, "fleet_type": "refrigerated_box"}
        for s in range(5, 69, 2)
    ]
    a = sweep_minor_major_boundary(_frame(rows))
    b = sweep_minor_major_boundary(_frame(rows))
    assert a == b  # identical input -> identical fitted boundary


def test_recall_weighting_favors_lower_threshold_on_ties():
    # Two thresholds with equal F-beta must resolve to the lower one.
    scores = pd.Series([30, 40, 50, 60])
    labels = pd.Series([False, True, True, True])
    lo = fbeta_at_threshold(scores, labels, 40, beta=2.0)
    assert lo["recall"] == 1.0  # catching every OOS defect is the goal

• Label leakage. If the ground-truth label is derived from the model’s own past routing decision — “we flagged it, therefore it was OOS” — the calibration learns to reproduce its prior output instead of reality, and the F-score looks excellent while predicting nothing. Source labels from independent downstream outcomes (actual OOS orders, roadside violations) and drop any row whose label_source is the model itself, as Step 1 does.
• Class imbalance. True OOS defects are rare, often well under a few percent of records. A boundary optimized for raw accuracy will call everything minor and still score 97% accurate while missing every OOS defect. Optimizing recall-weighted F-beta, not accuracy, is what forces the sweep to find the rare positives, and it is why beta = 2.0 rather than the balanced beta = 1.0.
• Overfitting to one season. A boundary fit on winter data absorbs winter tire-wear and cold-brake failure rates that do not hold in July. Calibrate across a full annual cycle or hold out a season for validation, and never let a single quarter set a fleet’s boundary — the tuning layer’s exponential smoothing exists precisely to damp this, but a raw calibration handed straight to production skips that protection.
• Silently moving the floor. The most serious failure is a calibration that improves the F-score by lowering where OOS-criteria defects land. The guardrail in Step 4 rejects this categorically: the sweep cannot propose a boundary at or above 70, and the floor check fails the whole run if any OOS-criteria defect falls below it. A better fit never justifies breaching § 396.11©(2).

• Dynamic Threshold Tuning for Fleet Types — consumes the empirical boundaries this calibration fits.
• Severity Scoring Algorithms for DVIR Defects — produces the 0–100 scores calibration is fit against.
• Building a Weighted Defect Scoring Model in Python — the deterministic scoring function whose output feeds the labeled dataset.
• Out-of-Service Criteria and OOS Hold Triggers — defines the OOS conditions the labels are built from.

Part of the Dynamic Threshold Tuning for Fleet Types guide. Back to Defect Classification & Repair Order Routing.