Skip to main content

Module 4: Scale, Reliability & Performance: Worked Examples

Example 1: Turn an SLO Into an Error Budget​

A 99.9% monthly availability SLO permits roughly 43.2 minutes of unavailability in a 30-day month. If the service spends 70% of that budget in five days, the decision is not to celebrate 99.93% current availability; release risk must be reduced while reliability work addresses the burn.

Use both short and long burn-rate windows so fast outages alert quickly without making every transient spike page an operator.

Example 2: Explain Queue Growth​

At 100 requests/s with 8 ms average service demand, utilization is 0.8 for one equivalent server. As arrival rate approaches service capacity, waiting time grows nonlinearly even before saturation. Measure percentiles and queue depth; averages hide tail pain. Add capacity or shed/defer load before the queue becomes the outage.

Example 3: Prevent a Retry Storm​

Three retries at every layer can amplify one request dramatically. Retry in one responsible layer, cap attempts by deadline and budget, add exponential backoff with jitter, honor server overload signals, and require idempotency for mutations.

Completion Standard​

  • Define SLIs/SLOs from user-visible outcomes.
  • Run a load test through saturation and explain the knee.
  • Demonstrate graceful overload and bounded retry behavior.