Skip to main content

Privacy, Data Minimization, and Memorization

What This Concept Is​

This lesson teaches you to limit collection, retention, exposure, and memorization while preserving deletion and incident response. Safety is a property of the complete socio-technical system: data, model, prompts, retrieval, tools, interfaces, operators, users, and incident controls. A benchmark score cannot stand in for that system evidence.

Why It Matters Here​

AI behavior is probabilistic, context-sensitive, and attractive to adversaries. Failures may be fluent, intermittent, unequally distributed, or visible only after an action is taken. The engineering response is layered control, adversarial evidence, bounded authority, monitoring, and accountable decisions.

Concrete Example​

Apply the concept to an assistant that retrieves internal policy and can create support refunds. Map trust boundaries, create a normal test and an adversarial variant, run both against a frozen system version, and trace every claim and action authorization. Preserve the failed example rather than rewriting it away.

Common Confusion / Misconception​

A common mistake is treating a system prompt, content filter, benchmark average, or human-review label as a complete control. Each can fail. State what the control prevents, what it merely detects, its bypass conditions, and the independent fallback that limits harm.

How To Use It​

  1. Name assets, actors, affected people, and prohibited outcomes.
  2. Map trust and privilege boundaries.
  3. Define measurable normal and adversarial claims.
  4. Test controls independently and in combination.
  5. Preserve failures with system and data versions.
  6. Assign owners, response thresholds, and expiry dates.
  7. Document residual risk and authorization to proceed.

Check Yourself​

  1. Which component is untrusted here?
  2. What is the highest-impact action reachable after failure?
  3. Which test distinguishes prevention from detection?
  4. Who can stop the system and on what evidence?

Mini Drill or Application​

Create five adversarial cases for this mechanism, including one indirect attack and one legitimate-but-unusual request. Run them through the complete system, classify first failure, repair one control, add a regression test, and record the residual risk and human fallback.

Read This Only If Stuck​