Skip to main content

Model Artifact: Runbook

Scenario​

A Semester 9 learner writes an on-call runbook for elevated API error rate.

Completed learner-quality example​

Alert​

APIHigh5xxRate: 5xx responses exceed 2% of requests for 10 minutes on the public API.

Severity​

Start as SEV-2 if more than one customer is affected. Escalate to SEV-1 if checkout, login, or data integrity is affected.

First five minutes​

  1. Acknowledge the alert.
  2. Open the API overview dashboard.
  3. Check whether errors are concentrated by endpoint, region, or deployment version.
  4. Start an incident channel if the error rate is still above threshold.
  5. Assign incident commander and scribe if more than one responder joins.

Diagnosis checklist​

  • Recent deployment? Compare error rate before and after release marker.
  • Dependency failure? Check database, cache, and payment provider panels.
  • Capacity issue? Check saturation: CPU, memory, connection pools, queue depth.
  • Bad input spike? Inspect structured error logs by exception type and route.

Mitigation options​

  • Roll back the latest deployment if errors correlate with release marker.
  • Disable the risky feature flag if failures are isolated to a flagged path.
  • Shed noncritical background work if database or worker pools are saturated.
  • Escalate to dependency provider status channel if failures originate externally.

Customer communication trigger​

Post status-page update if user-visible errors persist for 15 minutes or affect checkout/login at any duration above five minutes.

Recovery and follow-up​

Declare recovery only after error rate stays below 0.5% for 20 minutes. File a post-incident review with timeline, customer impact, root cause, mitigation, and prevention owner.

How to read this example​

  • Passing: Gives responders clear first actions, diagnosis paths, mitigation options, and recovery criteria.
  • Strong: Includes severity, communication triggers, and role assignment.
  • Portfolio-worthy: Makes the system operable by connecting alerts, dashboards, customer impact, and post-incident learning.