Give each probe one operational purpose
What This Concept Is
A probe is a control input with consequences. Startup gates initialization, readiness influences service traffic eligibility, and liveness can cause container restart. A green process check is not proof that a business operation succeeds.
Why It Matters Here
The existing rollout lab needs failure-specific probes. Restarting every instance because a shared dependency is unavailable can turn one dependency failure into a restart storm.
Concrete Example
A synthetic API can still answer its local liveness endpoint while its database is unavailable. Its write readiness may become false, while a separate cached-read operation remains usable. On a slow start, a startup probe prevents premature readiness/liveness evaluation until it succeeds. Test the actual failure thresholds and record events rather than assuming the configured period is an exact deadline.
Common Confusion / Misconception
Readiness does not restart the container. Endpoint propagation and existing connections are not instantaneous. A liveness check that depends on every downstream service can make healthy processes restart together.
How To Use It
Write each probe predicate, failure consequence and expected observation. Exercise slow start, local deadlock and dependency loss separately, recording restart count, Ready condition, EndpointSlice state and client-visible outcomes.
Check Yourself
Which failure should restart a process? What proves traffic stopped? Can a healthy endpoint hide a broken write path?
Mini Drill or Application
Before running, predict the three failures. Compare predictions to raw cluster events and request results, then repair one misleading probe.
Read This Only If Stuck
Building Secure and Reliable Systems provides selective operational reasoning. Use the module sources below for current tool behavior; do not read a whole book before attempting the exercise.