Backfill safely and prove coverage at cutover
What This Concept Is
Backfill transfers existing state while newer changes may be arriving. Version-aware merge prevents an older snapshot from overwriting a later value. Coverage is a separate property: every authoritative record and deletion relevant to the cutover must be represented.
Why It Matters Here
A migration can have correct per-key merge logic and still omit an entire partition. Row counts can match while values differ. A high-water marker indicates a position to verify, not proof that all changes up to that point were applied.
Concrete Example
For key x, old owner emits version 1 with value "a", version 2 with an empty string and version 3 with a deletion tombstone. Deliver them to the target in reverse order. The target must retain version 3, including the tombstone. Dropping the tombstone lets a delayed version 1 resurrect the record. Empty string is a stored value, not deletion.
Repeated identical version/value is harmless; equal-version different-value is a conflict. In this lab such a conflict rejects without mutation. A target with x at the right version but the wrong payload must not pass validation merely because source and target each have one row.
Common Confusion / Misconception
A zero lag metric may cover only the stream being observed; missing partitions or an incomplete snapshot can remain invisible. Comparing two actively changing stores can also produce misleading mismatches. Verification needs a common logical boundary and a declared definition of equality.
How To Use It
Record snapshot boundary, change-stream coverage and per-key version semantics. In the bounded lab freeze old writes, then require exact source/target equality including keys, versions and tombstones. Real large migrations need scalable range/checksum or reconciliation methods tied to a consistent boundary; sampling alone does not prove complete equality. Define how extra target keys are treated and how conflicts are investigated.
Check Yourself
Why do tombstones survive backfill? What does a matching row count fail to show? Why is a watermark insufficient by itself? What evidence ensures all source partitions were included?
Mini Drill or Application
Deliver stale and duplicate snapshots around a deletion. Omit one key and attempt cutover; add an extra target key and repeat. Preserve the refusal evidence. In the original database lab, compare actual values and deleted records at the chosen consistent boundary, not only counts.
Read This Only If Stuck
Microservices Patterns — service data and migration tradeoffs. Reuse Semester 06 replay and storage lessons for version/tombstone semantics, but add the authority and coverage proof needed for ownership transfer here.