Test secret rotation as a state transition

What This Concept Is

Rotation introduces a new credential, moves consumers, and revokes the old one. Each boundary can fail independently. Merely creating a second secret proves neither adoption nor revocation.

Why It Matters Here

The original secret-lifecycle lab requires continued traffic and audit evidence; these must show both legitimate access and denial after revocation.

Concrete Example

A worker caches credential A while the API adopts B. New API requests succeed, hiding a worker that will fail when A is revoked. Exercise both consumers before and after revocation. Record allowed identities and timing, never secret values. Repeated old-credential attempts must fail after the defined completion boundary while B still works.

Common Confusion / Misconception

A failed old request can be a network error rather than authorization enforcement. A positive control distinguishes a healthy service denying A from a broken service denying everyone.

How To Use It

Define issuance, propagation, overlap, revocation and recovery owners. Use synthetic credentials in a local service or authorized secret system. Inspect logs for accidental disclosure and verify the old value is absent from source/artifacts.

Check Yourself

What proves every consumer switched? Can revocation be tested with an unavailable server? How is overlap bounded?

Mini Drill or Application

Pause one consumer during rotation, resume it after revocation, and explain the observed recovery path. Preserve denied and allowed requests with redacted evidence.

Read This Only If Stuck

Building Secure and Reliable Systems provides selective operational reasoning. Use the module sources below for current tool behavior; do not read a whole book before attempting the exercise.

OpenTelemetry sensitive data; Collector security; Google SRE monitoring.