Connect alert evidence to measured containment
What This Concept Is
An alert is useful when it identifies a condition needing action and supports a response that reduces harm. The alert calculation, service saturation behavior and incident response are connected but require different evidence.
Why It Matters Here
A correct burn evaluator cannot show that admission control works, a dependency timeout contains failure, or a responder can recover the system. The original load and incident labs remain essential after the arithmetic passes.
Concrete Example
A dependency slowdown fills a queue. Request latency increases; retries add demand. A burn alert detects bad user outcomes but does not identify whether the bottleneck is CPU, a connection pool or the downstream service. Correlate queue depth, throughput, utilization and traces; apply a justified mitigation such as bounded admission or reduced concurrency and measure the resulting user impact.
If shedding requests lowers reported latency because rejected requests disappear from the SLI, the experiment is misleading. Retain offered load, rejection counts, completed throughput and the agreed error classification. Compare the same workload and report the cost of degradation.
Common Confusion / Misconception
A queue model does not produce measured p95/p99. A short successful test is not proof that the service survives arbitrary spikes or regional outages. A rollback can also fail or be incompatible with new data; use the previously tested migration constraints.
How To Use It
Preserve the original idle-to-overload experiment, containment change and repeated load test. Record environment, workload, client scheduling, raw latency samples, errors and queue/utilization signals. Run a local game day with a hypothesis, stop condition, role assignments and recovery procedure. Keep a timeline and verify that follow-up actions change an observed failure, not only a dashboard label.
Check Yourself
What evidence separates detection from diagnosis? Which metric can improve while users get more errors? How would you notice a load generator falling behind its intended offered rate? What confirms recovery beyond the alert clearing?
Mini Drill or Application
Inject dependency latency into the original service, detect it using your chosen alert rule, mitigate, and verify recovery with an authorized healthy request and an overload case. Record one alternative diagnosis and the evidence that ruled it out. Keep the incident review blameless and actions owned.
Read This Only If Stuck
Fundamentals of Software Architecture — reliability tradeoffs. Use the semester's SRE books selectively for overload and incident response; retain empirical evidence rather than importing default alert thresholds unchanged.