Quarantine and replay without rewriting history
What This Concept Is
Quarantine separates a message that cannot currently be processed from normal flow while retaining enough evidence to diagnose and recover. Replay is a controlled new delivery attempt under a declared policy, not automatic proof that the message is now safe.
Why It Matters Here
A message can be malformed, unsupported by the current consumer, semantically contradictory or merely failing because a dependency is unavailable. These cases need different responses. Replaying an unchanged poison message indefinitely only repeats the failure and can block healthy work.
Concrete Example
A payment success event is quarantined because a consumer cannot decode its new schema version. Preserve original bytes, identity, ordering key, schema version, failure reason, timestamps and restricted access. Repair the decoder and verify compatibility fixtures. Replay a small selected batch with stable business identities, rate limits and stop conditions. A new broker delivery ID must not become a new charge or refund identity.
If the payload itself was wrong, do not overwrite the historical event as though it had always been correct. Publish a governed correction or record an explicit transformation with provenance and tests. A later message may already have changed the aggregate; assess ordering and current state before replay.
Common Confusion / Misconception
Moving a message to a dead-letter queue is not successful business processing. Queue retention can expire evidence before investigation. Removing one message from an ordered stream can change application behavior; broker ordering does not repair a missing prerequisite or an obsolete business action.
How To Use It
Define quarantine ownership, access, retention, alerting and maximum unresolved age. Classify the failure before deciding retry, code repair, correction or escalation. Rehearse diagnosis, repaired fixture, bounded replay, duplicate detection and reconciliation against business state. Avoid exposing raw sensitive payloads in logs or a public portfolio.
Check Yourself
Which identity survives redelivery? When should a payload correction become a new event? Can healthy traffic overtake a quarantined event safely? What evidence is lost if quarantine retention expires?
Mini Drill or Application
In the original broker/consumer lab, quarantine an unsupported event, process unrelated work, repair the consumer and replay with a controlled rate. Inject a duplicate and a replay after later state changes. Preserve the operator record, before/after business state and stop-condition test.
Read This Only If Stuck
Microservices Patterns — event contracts and operations. Amazon SQS dead-letter guidance provides one broker-specific example of retention and ordering considerations. Use your chosen broker's actual semantics; no cloud account is required for this local exercise.