Compare tactics and preserve the limits of evidence
What This Concept Is
A tactic experiment tests a proposed mechanism against the same scenario as its alternative. A change in a metric is evidence about that experiment, with specific workload, data and failure conditions. It is not automatically a causal explanation or a universal architecture ranking.
Why It Matters Here
A cache can reduce response time while serving deleted data. A replica can spread reads while violating read-your-writes. These are tradeoffs only when the relevant correctness and quality conditions are visible; hiding one in an average prevents an informed decision.
Concrete Example
Candidate A serves 100 good attempts out of 100 for cohort X and zero of four for cohort Y: pooled 100/104 exceeds 95%, while Y gets no useful service. Candidate B serves 97/100 and 4/4: pooled 101/104 and both cohorts improve relative to their respective constraints except X. Decide using requirements established before results, not an arbitrary pooled ranking. If the target for X is 99%, B fails that target despite its stronger pooled score.
For a cache-versus-direct-read experiment, run both over the same synthetic documents and request trace. Record cold and warm states separately. Include update, deletion and expired-permission reads. Record cache memory, invalidation work and observed stale results, not only latency. A zero-leakage requirement remains a separate gate.
Common Confusion / Misconception
An import restriction is a proxy for one change boundary, not proof that changes are easy. Time a concrete change and inspect tests and deployment impact. A secret-pattern scanner cannot demonstrate that all sensitive data is absent; an unavailable log query must not be interpreted as a clean scan.
How To Use It
Freeze workload, candidate versions, environment, thresholds and cohorts. Repeat each candidate in interleaved order, preserve raw observations and report variation. Hold one failure trace back until the candidate is chosen. Record a rejected alternative, the weakest assumption and a trigger for reopening the decision. If a counterexample violates an invariant, repair it before accepting a speed benefit.
Check Yourself
Which population is hidden by pooled success? How might warm-up order bias results? Does a no-cycle dependency graph establish maintainability? What is the smallest experiment that could falsify your favored tactic?
Mini Drill or Application
Run the original Tactic Experiment with two implementations and the evaluator. Add an independent integrity check. Write a one-page decision naming both gains and costs, uncertainty and a reversal trigger. Draw where measurements occur on the runtime and deployment views. Keep the other original architecture views and stakeholder work.
Read This Only If Stuck
Fundamentals of Software Architecture — tradeoffs and fitness functions. Revisit Semester 06 replication and outbox exercises when the tactic changes freshness or retry behavior.