Make canary decisions with sufficient exposure

What This Concept Is

A canary compares candidate behavior with a defined baseline and abort policy. Low traffic or missing telemetry can leave the decision inconclusive. A green deployment status is not sufficient exposure.

Why It Matters Here

Semester 08 burn windows already handle missing data. Apply the same honesty to release decisions without copying its evaluator as a new release system.

Concrete Example

Baseline has 1 error among 100 requests; candidate has 0 among 2. Zero observed errors is not persuasive evidence of improvement. Candidate requests may also be easier. Define eligible requests, exposure minimum, comparison window, guardrails and a maximum observation period before rollout. On missing collection, hold or abort according to the recorded policy instead of silently promoting.

Common Confusion / Misconception

An inconclusive result is neither demonstrated failure nor demonstrated safety. Shared dependency regressions can affect both groups and evade a relative comparison; keep absolute guardrails.

How To Use It

Preserve counts and traffic assignment. Inject one candidate regression, one shared failure and collector loss. Demonstrate actual promotion/hold/abort behavior and an operator escalation path.

Check Yourself

What does zero of two establish? Which failures affect both cohorts? What happens when the observation deadline arrives without evidence?

Mini Drill or Application

Replay the three scenarios in the original progressive-delivery exercise. Record the decision before examining the final result and explain each deviation.

Read This Only If Stuck

Building Secure and Reliable Systems provides selective operational reasoning. Use the module sources below for current tool behavior; do not read a whole book before attempting the exercise.

SLSA v1.2 verification; Google SRE canarying; DORA metric definitions.