The Four Golden Signals: Observability That Prevents Alert Fatigue
Why CPU alerts wake engineers up at 3 AM for harmless spikes, and how restructuring alerting around Latency, Traffic, Errors, and Saturation creates actionable, high-signal alerts.
Every software engineering team eventually experiences the dread of alert fatigue. The Slack channel dedicated to monitoring receives dozens of notifications every day: CPU usage hit 82% for two minutes, memory reached a threshold, or a cron job finished 10 seconds later than usual.
Because 99% of these alerts require no action and resolve themselves, engineers quickly learn to ignore them. Then, when a genuine critical failure strikes—such as checkout requests silently failing for all mobile users—the real alert gets lost in the noise.
Breaking out of alert fatigue requires adopting the Four Golden Signals of Monitoring, popularized by Google Site Reliability Engineering (SRE).
The Four Golden Signals
Instead of monitoring low-level machine metrics that correlate poorly with user experience, monitoring should focus strictly on symptoms that impact system health:
1. Latency
The time it takes to service a request. It is critical to differentiate between the latency of successful requests and the latency of failed requests (e.g. an immediate HTTP 500 due to a missing database connection is fast, but masks slow degraded queries).
- Rule: Never alert on average latency. An average hides critical tail latency spikes. Always measure P95 and P99 percentiles.
2. Traffic
A measure of how much demand is being placed on your system. For a web service, this is typically requests per second (RPS); for a message consumer, it is messages processed per minute.
- Rule: Traffic drops are often more critical than traffic spikes. A sudden 70% drop in traffic usually indicates an upstream DNS or ingress routing failure.
3. Errors
The rate of requests that fail, either explicitly (HTTP 500s), implicitly (HTTP 200 containing an error payload), or by policy (e.g., requests exceeding a 5-second deadline).
- Rule: Alert on error percentage rather than absolute error counts. 10 errors per minute during peak traffic (10,000 RPS) is normal background noise (0.1%); 10 errors per minute during a 3 AM low (15 RPS) is a 66% failure rate.
4. Saturation
How "full" your service is, measured by the resource with the most constrained bottleneck (e.g. database connection pool exhaustion, memory usage on a Redis cache, or disk I/O queue depth).
- Rule: Saturation alerts should be predictive. Alert when a resource is on a trajectory to exhaust capacity within an hour, giving engineers time to react before user impact occurs.
| Signal | Bad Alert (Noisy / Inactionable) | Good Alert (High Signal / Actionable) |
|---|---|---|
| Latency | "Host API-01 CPU > 80%" | "P99 checkout latency > 1.8s for 5 consecutive minutes" |
| Traffic | "Requests per second changed by 10%" | "Traffic dropped > 50% compared to same hour last week" |
| Errors | "5 errors detected in log stream" | "5xx HTTP error rate exceeds 1.5% of total requests over 3m" |
| Saturation | "Server memory at 85%" | "Database connection pool utilization > 92% for 4 minutes" |
Designing Actionable Runbooks
An alert without an immediate action is not an alert; it is a dashboard metric.
Every pager alert that interrupts an engineer should satisfy three criteria:
- Direct User Impact: Does this failure actually degrade the experience for customers right now?
- Actionability: Is there an immediate step an on-call engineer can take to mitigate or diagnose the problem? If the answer is "no, wait for it to clear," delete the alert.
- Linked Runbook: Every alert notification must include a direct link to an actionable runbook detailing:
- What the alert means.
- The primary dashboards to inspect.
- The exact mitigation steps (e.g., rolling back a deployment, scaling up a worker replica, or enabling a circuit breaker).