Finish or safely retry work during termination
What This Concept Is
Graceful termination is a bounded protocol: stop accepting new work, finish or relinquish in-flight work, then exit. A timeout ends the grace period; it does not prove that unfinished work was harmless.
Why It Matters Here
A successful rollout command can conceal lost or duplicated jobs. The worker/database deployment from the original lab needs a business invariant through shutdown.
Concrete Example
Give a synthetic job an operation ID. Terminate the worker after its database effect but before acknowledgement. On retry, the consumer must avoid a second business effect. Record an attempt log separately from the committed result. A shorter grace period can expose the same cut even when normal shutdown appears clean.
Common Confusion / Misconception
A termination hook is not a transaction. Exactly-once business behavior needs durable identity/atomicity where appropriate; a long sleep does not provide it. StatefulSet identity also does not supply database replication automatically.
How To Use It
Reuse the Semester 06 outbox/idempotency evidence. Trace intake, effect, acknowledgement and shutdown boundaries. Test both completed and interrupted work, and verify data after restart.
Check Yourself
Which event permits acknowledgement? What if termination happens after the effect? How do you distinguish a retry from a new job?
Mini Drill or Application
Submit one job while terminating a worker, repeat at a different boundary, and show the result invariant plus all attempts. Keep client timestamps and controller events.
Read This Only If Stuck
Building Secure and Reliable Systems provides selective operational reasoning. Use the module sources below for current tool behavior; do not read a whole book before attempting the exercise.