Tuning Alert Thresholds to Reduce Alarm Fatigue

The exact task this guide solves is setting the warning thresholds and stabilising logic so a duty-time engine alerts schedulers to genuine risk without burying them in near-miss noise. An engine that fires on every value within a minute of a limit trains schedulers to dismiss alerts, and a dismissed alert is worse than none because it hides the real violation. The fix is not a single number but a tuning discipline: a buffer band between “fine” and “hard limit”, hysteresis so a value hovering on the edge does not flap, and a measured trade-off between catching real problems and crying wolf. This page builds that logic and shows how to tune it against history. It is the operational core of the threshold tuning and alerting section and consumes verdicts from the rest period compliance checks and the fatigue layer.

Prerequisites

Why a single threshold is the wrong tool

A hard regulatory limit is a cliff: below it legal, above it a violation. But schedulers need warning before the cliff, and a naive warning threshold set a few minutes inside the limit fires constantly on rosters that are tight but fine. Two mechanisms fix this. A buffer band defines a zone approaching the limit where the engine warns rather than blocks, sized so it captures duties worth a second look without flagging every tight-but-normal day. Hysteresis requires a value to move meaningfully back inside the band before an alert clears, so a margin oscillating around the edge does not produce a stream of raise/clear churn.

Step 1 — Define the buffer band

The band sits between a warning threshold and the hard limit. A margin comfortably inside the band is silent; a margin in the band warns; a margin past the limit is a violation. Expressing all three as data lets compliance tune them without a deploy.

from dataclasses import dataclass
from enum import Enum


class AlertLevel(Enum):
    OK = "ok"
    WARN = "warn"
    VIOLATION = "violation"


@dataclass(frozen=True)
class Band:
    warn_margin_minutes: int   # start warning when within this of the limit
    hard_limit_minutes: int    # the regulatory ceiling


def classify_margin(minutes_to_limit: float, band: Band) -> AlertLevel:
    if minutes_to_limit < 0:
        return AlertLevel.VIOLATION
    if minutes_to_limit <= band.warn_margin_minutes:
        return AlertLevel.WARN
    return AlertLevel.OK

Verify: with a 30-minute warn margin, a duty 45 minutes inside its limit is OK, one 20 minutes inside is WARN, and one 5 minutes over is VIOLATION — the band captures the approach without flagging the comfortable case.

Step 2 — Add hysteresis so alerts do not flap

A margin drifting across the warn threshold on successive schedule revisions should not raise and clear repeatedly. Hysteresis uses two thresholds: an alert raises at the warn margin but only clears once the margin recovers past a wider clear margin.

def apply_hysteresis(prev_level: AlertLevel, minutes_to_limit: float,
                     band: Band, clear_margin_minutes: int) -> AlertLevel:
    raw = classify_margin(minutes_to_limit, band)
    if prev_level in (AlertLevel.WARN, AlertLevel.VIOLATION) and raw is AlertLevel.OK:
        # Only clear once comfortably back inside the band.
        if minutes_to_limit < clear_margin_minutes:
            return AlertLevel.WARN
    return raw

Verify: an alert raised at a 20-minute margin does not clear when the margin recovers to 32 minutes if the clear margin is 45; it clears only once the margin exceeds 45, so a value hovering near 30 stays stable instead of flapping.

Step 3 — Tune the band against labelled history

The band width is a precision/recall trade-off: a wide warn margin catches more real issues (higher recall) at the cost of more false alarms (lower precision). Sweeping the margin against a labelled history and reading off precision and recall lets compliance pick a defensible operating point rather than guessing.

def score_threshold(history: list[dict], warn_margin: int) -> dict:
    """history rows: {'margin': float, 'was_real_issue': bool}."""
    band = Band(warn_margin_minutes=warn_margin, hard_limit_minutes=0)
    flagged = [h for h in history if classify_margin(h["margin"], band) is not AlertLevel.OK]
    true_pos = sum(1 for h in flagged if h["was_real_issue"])
    total_real = sum(1 for h in history if h["was_real_issue"])
    precision = true_pos / len(flagged) if flagged else 0.0
    recall = true_pos / total_real if total_real else 0.0
    return {"warn_margin": warn_margin, "precision": round(precision, 3), "recall": round(recall, 3)}

Verify: sweeping warn_margin from 10 to 60 minutes produces a curve where recall rises and precision falls; the chosen margin is the knee where recall is high enough to catch real issues before precision collapses into alarm fatigue.

Alert band: OK, warn, and violation zones with hysteresis A horizontal margin axis from comfortable on the left to the hard limit on the right. A green OK zone covers large margins, an amber warn band covers the approach to the limit, and a red violation zone lies past the limit. A hysteresis gap shows that an alert raised in the warn band only clears once the margin recovers well into the OK zone. OK comfortable margin WARN band approaching limit VIOL. large margin hard limit → alert clears only here — hysteresis gap
The warn band captures the approach to the hard limit; hysteresis holds an alert until the margin recovers well into the OK zone, so a value hovering near the edge does not flap.

Failure modes and troubleshooting

Frequently Asked Questions

Why warn before the hard limit at all?

Because a scheduler needs time to rebuild a pairing before publication. A warning band gives lead time on duties approaching a limit, while the hard limit remains the block. Without the band, the first signal is the violation, which is too late to fix cheaply.

How wide should the warn band be?

Wide enough to catch duties worth reviewing, narrow enough not to flag every tight-but-normal day. The defensible way to choose is to sweep the margin against labelled history and pick the operating point where recall is adequate and precision has not collapsed.

What problem does hysteresis solve?

Flapping. A margin oscillating around the warn threshold across schedule revisions would otherwise raise and clear an alert repeatedly. Hysteresis holds the alert until the margin recovers comfortably, so the signal is stable.

Is alarm fatigue really a safety issue?

Yes. When schedulers learn to dismiss a noisy alert stream, they dismiss the real violation with it. Reducing low-value alerts is what keeps the high-value ones visible, so tuning is a safety control, not just a convenience.

Back to Threshold Tuning & Alerting.