Skip to main content

Privacy-Aware Data and Retrieval Systems

Competency Contract​

  • Primary competency: DM-DATA-001
  • Mastery: explain, apply, develop, and evaluate as required by the semester assessment
  • Status: required evidence extension; budget 35–50 hours inside the reconciled semester workload

Required Work​

Design collection, lineage, retention, deletion, and access around purpose; build a streaming or change path; evaluate vector or ranked retrieval separately from downstream generation.

Evidence Contract​

  • data inventory and lineage
  • retention/deletion test
  • stream replay evidence
  • retrieval benchmark and failure analysis.

Assessment Integration​

  • Project: produce the evidence in the semester repository rather than as detached prose.
  • Checkpoint: show the evidence exists and identify its highest-risk gap.
  • Cumulative review: connect the work to current, previous, and two-semesters-back knowledge.
  • Exam: evaluate a new case and reject an attractive but unsafe or unsupported alternative.
  • Rubric: every required evidence category must pass; strengths cannot average away a missing category.
  • Answer key: calibrate the shape of evidence without prescribing one implementation.

Oral Defense​

Trace one record through collection, replication, deletion, and retrieval, including a privacy failure.

AI-Use Declaration​

Record tools and models used, material output accepted, rejected suggestions, and independent verification. Undeclared generated evidence fails the integrity gate.

Reading Route​

Start with the linked semester concepts and labs. Use local books selectively for conceptual depth and current official standards for normative requirements. Stop reading when you can produce and defend the required evidence.

One synthetic data path​

Use a small tenant-scoped document service with a source table, change/outbox stream, search projection and optional cache. Choose ranked keyword retrieval as the baseline; vector retrieval is optional, not required infrastructure. Keep retrieval evaluation separate from generated answers. Start with at least 30 synthetic documents across two tenants and a frozen set of 12 query cases, including no-answer and access-denied cases.

D1 — Inventory and lineage (6–8 hours)​

For each field, record purpose, source, owner, access rule, transformations, copies and deletion path. Include source rows, outbox payloads, replica copies, indexes, caches, logs and backups. Draw one record's path and list derived fields such as token sets or embeddings. “Deleted from the source table” is not evidence that all copies disappeared. For every retained copy, name its purpose and bounded retention policy; distinguish logical inaccessibility from physical erasure.

D2 — Retention, access and deletion (8–12 hours)​

Create two tenants with deliberately overlapping vocabulary. Test permitted and denied source reads and retrieval, then delete a target document. Assert it no longer appears in direct lookup, keyword/vector results or cached responses accessible to that tenant. Test an expired record separately from an explicit deletion. Apply access filtering before returning results, and record whether filtering after top-k truncation loses eligible results.

State the deletion completion point: for example, all online serving paths deny the document and asynchronous cleanup is tracked separately. Preserve only the minimum deletion identity/version needed for replay protection and explain its own retention. Restore a disposable older backup and show the deletion policy is reapplied before the restored service becomes available. Do not claim physical media erasure or legal compliance from these application tests; document inaccessible backups and their expiry assumptions explicitly.

D3 — Stream replay and resurrection prevention (9–13 hours)​

Reuse the outbox/inbox lessons to project versioned updates into retrieval state. Use stable event IDs and monotonically increasing versions per document. Duplicate events must not duplicate entries; an older update must not overwrite a newer delete. Define the equal-version/different-payload conflict outcome. Retain a deletion marker or equivalent policy until older sources of resurrection can no longer be replayed.

Required trace: create v1, update v2, delete v3, delayed v1, duplicate v3. Final serving state must remain absent. Rebuild a fresh projection from the declared source and replay log, then compare state with the live projection. Repeat with a snapshot taken before deletion. If snapshot, log and deletion ledger have incompatible retention, record a failing recovery case rather than silently discarding it. Distinguish delivery deduplication from version-based stale-update rejection: event IDs alone do not order different updates.

D4 — Retrieval benchmark and failure analysis (12–17 hours)​

Freeze corpus, tenant, query text, expected eligible relevant IDs and k=3 before tuning. Separate eight development cases from four held-out cases. Use explicit relevance judgments; explain ambiguous labels. Compare a simple keyword baseline with one justified change, such as phrase handling or a vector index. Do not add a paid model merely to create a second baseline.

For answerable queries, define precision@3 as relevant eligible results in the first three slots divided by 3, including empty slots, and recall@3 as that count divided by the number of eligible relevant documents. Report per-query results and macro averages over answerable cases. For zero-relevant queries, recall is undefined: report the number of such cases and an incorrect-result rate separately. Measure unauthorized-result count independently; any unauthorized result fails the access gate even with excellent average relevance.

Worked calibration: relevant IDs are A and C; returned ranking is A,B,D. Precision@3 is 1/3 and recall@3 is 1/2. Repeating A twice must not earn two relevant hits: require unique document IDs. If C has been deleted, update the evaluation snapshot and its relevance set explicitly; do not score against stale labels unknowingly. Publish corpus/query versions, ranking output and a short false-positive/false-negative analysis. Keep generated-answer fluency out of these retrieval scores.

Calibration and advancement​

Each D1–D4 requires level 2 under the workload/progression contract. A diagram without tested lineage, a deletion claim covering only one table, a replay demo without a stale update, or a single average retrieval score without access checks is below passing. Strong evidence includes a failed attempt, minimal reproducer, repair and changed-case retest.

In the semester defense, trace a privacy failure through collection, replication, deletion and retrieval. After 48 hours, change the stale-event version or tenant and solve independently. Preserve tool/AI assistance and identify which outputs were actually executed. No synthetic/generated output may be presented as an observed service trace.

Selective resources​

  • NIST Privacy Framework: use the inventory and risk-management perspective to identify processing purpose and exposure. This engineering exercise is not a compliance certification.
  • Introduction to Information Retrieval: ranked evaluation: consult the evaluation discussion when defining your benchmark and interpreting averages.
  • Continue DDIA's derived-data and replication selections for replay; use the semester's database texts for transactions and storage. No additional purchase or full-book conversion is necessary.