How to Stop Your Monitoring From Crying Wolf

Why false alarms are worse than no alarms

A team with no monitoring knows it is blind and behaves cautiously. A team whose alerts are 80% noise believes it is covered while having trained itself to dismiss notifications within seconds of seeing them.

The second team will miss a real outage, and they will miss it in a specific, predictable way: someone will glance at the alert, assume it is the usual noise, and go back to sleep.

The main sources of noise

• Single-probe failuresOne transient network hiccup between one prober and your origin. By far the largest category.
• Expired probe credentialsA token in the monitor config expires and every check starts returning 401. Looks like an auth outage, is not.
• WAF and bot filteringYour security layer starts blocking the monitoring source, producing 403s that have nothing to do with service health.
• Thresholds set too tightA latency budget at exactly today's p95 means normal variance trips it several times a week.
• FlappingAn endpoint oscillating between healthy and broken generating a stream of up/down pairs.

Cross-region confirmation

The single highest-impact fix. When a probe fails, do not change state — immediately re-check from other regions. Only declare an outage when independent vantage points agree.

This removes essentially the entire first category of noise, because a transient problem on one network path will not reproduce from a different continent. It costs a few seconds of additional detection time and eliminates most of your false pages.

Flap detection

An endpoint that fails, recovers, fails again, and recovers within a few minutes is genuinely unhealthy — but sending four alerts about it is worse than sending one. Collapse rapid state changes into a single ongoing incident and notify once, with a note that the service is unstable.

Setting thresholds with headroom

If your p95 latency is 400ms, do not set your alert at 400ms. Normal daily variance will breach it regularly. Set it somewhere around 700ms — high enough that routine fluctuation is quiet, low enough that a genuine regression trips it.

The same logic applies to check intervals and timeout values. Every threshold set at exactly the current normal is a false alarm generator.

Audit your alerts monthly

Look at every alert that fired in the past month and sort them into two piles: those that required action, and those that did not. The second pile is your backlog.

• Fix the underlying cause where you can — renew that credential, allowlist that probe
• Loosen thresholds that were set too aggressively
• Downgrade genuinely informational alerts to a low-urgency channel
• Delete monitors nobody acts on — an alert nobody responds to is not monitoring