How to reduce alert fatigue in Prometheus without losing real alerts

The symptoms (you probably have several)

• Pages get acknowledged from bed without being read. The on-call has learned that HighMemoryUsage at 03:00 means nothing — and acks it on autopilot.
• Slack alert channels are muted. Not filtered — muted.
• There's tribal knowledge like "ignore the disk alert on the batch nodes on Sundays". The routing didn't encode it; a human's patience did.
• New on-call engineers ask "is this one real?" and the honest answer is a shrug.
• Nobody dares delete or retune a rule, because nobody remembers why it exists and everyone fears the one time it would have mattered.

The last two are the dangerous ones. The industry observation is blunt: after enough false alarms, the team starts ignoring the warnings (AssetWatch). At that point your alerting system still costs money and sleep — it just no longer provides detection. It is functionally dead, and the outage that proves it is already scheduled; you just don't know the date.

What it costs, in numbers you can show your manager

Industry figures, sources in brackets:

• Static thresholds — which is what most Prometheus rule sets are — run at 20–40% false positives; well-calibrated baselines bring that to 5–15% [openobserve, AIOps guide].
• Each dispatched false positive costs an engineer 45–90 minutes: context switch, investigation, the "it's nothing again" write-up, and the climb back into whatever they were doing [reliamag/oxmaint].
• At a typical noisy-setup volume of 12–18 false alerts/day, a team of three loses up to 27 hours per week [reliamag/oxmaint]. That's most of a full-time engineer, permanently assigned to investigating nothing.

And the costs the spreadsheet doesn't capture: every false page erodes the credibility of the next real one, off-hours pages tax sleep and retention, and on-call dread is consistently one of the reasons platform engineers quit. If your team has had a departure where the pager was mentioned in the exit conversation, you already know this isn't hypothetical.

A triage framework: noisy / ok / insensitive

You can't fix a 300-rule set rule-by-rule on vibes. Triage first. Pull 90 days of alert history (the ALERTS metric in Prometheus, or your Alertmanager/paging-tool logs) and put every rule that fired into one of three buckets:

Two notes. First, the "insensitive" bucket is why pure alert-reduction crusades are dangerous: if your only metric is "fewer pages", you'll happily make rules blind. Audit both directions — replay your last five incidents against the rule set every time you tune. Second, expect a power-law: in most setups, 5–10 rules generate 60–80% of the noise. Triage means you fix those first and ignore the long tail until next quarter.

Quick wins in Alertmanager (this week, config only)

These don't fix bad thresholds, but they cut dispatched noise immediately:

• Grouping. Make sure group_by matches how incidents actually arrive. A node failure that pages once per pod is 40 pages for one event; grouping by cluster, alertname (or by service) makes it one notification with 40 members. Tune group_wait (e.g. 30s) so near-simultaneous alerts batch instead of trickling.
• Inhibition. Encode causality: when NodeDown fires, suppress the per-pod and per-target alerts on that node with an inhibit_rules entry. When the database is down, the "API error rate high" alert is telling you nothing the database alert didn't. Most teams have zero inhibition rules and pay for it in cascade pages.
• Severity-honest routing. If a rule's firings never lead to immediate action, it shouldn't reach the pager. Route severity: warning to a queue reviewed in business hours; reserve the pager for critical. Demoting a rule is often politically easier than deleting it — and operationally almost as good.
• Time-based muting for known patterns. If the batch cluster legitimately saturates every night at 02:00, a mute_time_intervals window is honest config. A human "everyone knows to ignore that one" is not.

The fix underneath: calibrate the thresholds

Everything above manages the noise after the rules fire. The root cause is that the thresholds fire wrongly in the first place — they were guessed, or set with point statistics (a 7-day p95, a mean+2σ) that don't model how metrics actually behave: autocorrelated, seasonal, and paging on sustained crossings rather than single samples. We covered the mechanics — and the false-positive-budget-per-rule concept that makes tuning decisions defensible — in the threshold-tuning guide.

The short version: for each rule, decide how many false pages per year it's allowed to cost, then derive the lowest threshold whose expected false-page rate — estimated against your own healthy history, with time dependence preserved — fits that budget. Do that, and the Alertmanager layer above goes back to being polish on a sound foundation instead of a tourniquet on a guessed one. The 20–40% → 5–15% false-positive improvement cited at the top is precisely the gap between guessed thresholds and calibrated ones.

Sequence that works in practice:

1. Week 1: triage (noisy/ok/insensitive) from 90 days of history. Grouping + inhibition pass in Alertmanager.
2. Week 2–3: recalibrate the top 5–10 noisy rules against healthy history; replay past incidents to verify detection; demote what shouldn't page.
3. Quarterly: re-triage. Thresholds decay as systems change — fatigue creeps back if tuning isn't a loop.